Ordinary Least Squares & Closed-Form Normal Equation | Foundations of Machine Learning

Ordinary Least Squares Normal Equation Lab

Machine LearningLesson / Article Lab

Compute analytical weights via (X^T X)^(-1) X^T y and compare against iterative optimization.

Ordinary Least Squares Closed-Form Normal Equation

Pyodide WasmCPython 3.12 • NumPy

Interactive Challenge: Modify code inputs, click Run Code to execute live in WebAssembly.

+25 XP Reward

Wasm Terminal Output

Click Run Code to execute this algorithm in the browser sandbox.

For linear models, the closed-form analytical solution that minimizes Mean Squared Error is given by the Normal Equation:

$\theta = (X^T X)^{-1} X^T y$

import numpy as np

# Generate synthetic linear data
np.random.seed(42)
X = 2 * np.random.rand(100, 1)
y = 4 + 3 * X + np.random.randn(100, 1)

# Add bias column (x_0 = 1)
X_b = np.c_[np.ones((100, 1)), X]

# Compute normal equation
theta_best = np.linalg.inv(X_b.T.dot(X_b)).dot(X_b.T).dot(y)
print(f"Learned Intercept: {theta_best[0][0]:.4f}")
print(f"Learned Slope:     {theta_best[1][0]:.4f}")