Gradient Descent, Convex Optimization & Learning Rates

Mathematical derivation of parameter updates, learning rate schedules, and momentum.

Interactive LabConvex & Non-Convex Optimization

Visualize how the learning rate $\eta$ and momentum influence convergence toward the global minimum.

Step 0 of Iteration

Weight (w)2.400

Loss J(w)6.157

Gradient ∇J5.713

Learning Rate (η)0.15

0.02 (Slow)0.15 (Ideal)0.95 (Unstable / Exploding)

Initial Starting Point (w₀)2.40

Momentum (β = 0.9)Helps escape plateaus and smooths oscillation

Gradient Descent iteratively nudges model parameters $\theta$ opposite to the gradient vector of the empirical risk function $J(\theta)$ :

$\theta_{t+1} = \theta_t - \eta \nabla_\theta J(\theta_t)$

Where $\eta$ is the learning rate. We explore Batch, Mini-Batch, and Stochastic variants alongside momentum techniques.

Machine LearningMatched to lesson

Simulates SGD with momentum navigating a 1D loss landscape.

Labs:

Gradient Descent Optimization Loop with Momentum

Python 3.13 • NumPy • PyTorch

Terminal Output

Click Run Code to execute this algorithm in the browser sandbox.

Mark it as complete to record your progress and unlock the next module.