Foundations of Machine LearningGradient Descent, Convex Optimization & Learning Rates
Video Lesson22 minutes

Gradient Descent, Convex Optimization & Learning Rates

Lesson Summary: Mathematical derivation of parameter updates, learning rate schedules, and momentum.

Gradient Descent Optimization & Learning Rate Lab

Machine LearningLesson / Article Lab

Compute partial derivatives dMSE/dm and dMSE/dc iteratively to converge on optimal parameters.

Gradient Descent Optimization Loop with Momentum
Pyodide Wasm
Interactive Challenge: Modify code inputs, click Run Code to execute live in WebAssembly.
+25 XP Reward
Wasm Terminal Output

Click Run Code to execute this algorithm in the browser sandbox.

Gradient Descent iteratively nudges model parameters θ\theta opposite to the gradient vector of the empirical risk function J(θ)J(\theta):

θt+1=θt−η∇θJ(θt)\theta_{t+1} = \theta_t - \eta \nabla_\theta J(\theta_t)

Where η\eta is the learning rate. We explore Batch, Mini-Batch, and Stochastic variants alongside momentum techniques.

Previous
Ordinary Least Squares & Closed-Form Normal Equation
Next Lesson
Ridge, Lasso, ElasticNet & The Bias-Variance Tradeoff