Skip to content

GRADIENT DESCENT PLAYGROUND

Explore 2D & 3D visuals, tune learning rate, and build intuition interactively.

What is Gradient Descent?

Gradient descent is a fundamental algorithm to find the minimum of a function. Imagine you're lost in a foggy valley: you can't see the landscape, but you can feel the slope beneath your feet. To reach the lowest point, you just keep stepping in the direction where the ground drops the fastest.

Mathematical Definition

We want to minimize a function f(x)f(\mathbf{x}), with x∈Rd\mathbf{x}\in\mathbb{R}^d.

xn+1=xn−η ∇f(xn)\mathbf{x}_{n+1} = \mathbf{x}_n - \eta \, \nabla f(\mathbf{x}_n)
  • xn\mathbf{x}_n: current position
  • ∇f(xn)\nabla f(\mathbf{x}_n): the gradient — the vector of partial derivatives
    ∇f=(∂f∂x1, ∂f∂x2, …, ∂f∂xd)\nabla f = \left(\frac{\partial f}{\partial x_1},\,\frac{\partial f}{\partial x_2},\,\dots,\,\frac{\partial f}{\partial x_d}\right)
  • η\eta: the learning rate controlling step size
Example: f(x,y)=x2+y2f(x,y)=x^2+y^2
∇f(x,y)=(2x,2y)\nabla f(x,y)=(2x,2y)
xn+1=xn−2η xn=(1−2η) xnyn+1=yn−2η yn=(1−2η) yn\begin{aligned} x_{n+1} &= x_n - 2\eta\,x_n = (1-2\eta)\,x_n \\ y_{n+1} &= y_n - 2\eta\,y_n = (1-2\eta)\,y_n \end{aligned}
Learning rate & stability
For this quadratic, iterations are stable when ∣1−2η∣<1|1-2\eta|<1 i.e. 0<η<10<\eta<1. The factor 1−2η1-2\eta tells the whole story:
  • 0<η<0.50<\eta<0.5: smooth convergence, xnx_n keeps its sign;
  • η=0.5\eta=0.5: the minimum is reached in a single step;
  • 0.5<η<10.5<\eta<1: zig-zags around 0 but still converges;
  • η≥1\eta\ge 1: never converges (it bounces at η=1\eta=1, diverges beyond).
Here L=2L=2, so the threshold is 2/L=12/L=1. More generally, for an LL–smooth function, any 0<η<2/L0<\eta<2/L guarantees a monotonic decrease of ff, and η=1/L\eta=1/L gives the largest guaranteed decrease.

How does it work, practically?

At each step, compute the gradient at your position; it points toward the fastest increase. To minimize, move in the opposite direction. Repeat until you’re “close enough” to a minimum (where the gradient vanishes).

Stopping criteria
  • ∥∇f(xn)∥≤εgrad\|\nabla f(\mathbf{x}_n)\| \le \varepsilon_{\text{grad}}
  • ∣f(xn+1)−f(xn)∣max⁡(1, ∣f(xn)∣)≤εrel\frac{|f(\mathbf{x}_{n+1}) - f(\mathbf{x}_n)|}{\max(1,\ |f(\mathbf{x}_n)|)} \le \varepsilon_{\text{rel}}
  • n≥nmax⁡n \ge n_{\max} (max iterations)
When does it work well?
  • Convex or well-conditioned landscapes
  • LL–smooth gradients
  • Proper step size η\eta
Common variants
  • Batch / mini-batch / SGD
  • Momentum / Nesterov
  • Adaptive: AdaGrad, RMSProp, Adam
Summary:Gradient descent is a simple, universal optimizer: always step “downhill” (the direction of minus the gradient). Choose a sensible η\eta and stop when the gradient is small.

Gradient Descent — 2D Views

We start with functions of a single variable f: R ⁣→ ⁣Rf:\ \mathbb{R}\!\to\!\mathbb{R}. Use the selector and the learning rate η\eta to see how gradient descent behaves on different 1D landscapes.

We use three classic 1D landscapes:

  • Quadratic — f(x)=x2f(x)=x^2🏔️ A simple bowl shape - there's only one bottom point. Easy for gradient descent to find!
  • Double well — f(x)=x4−x2f(x)=x^4-x^2🏔️⚡🏔️ Two valleys with a hill between. Where you start determines which valley you end up in!
  • Wavy quadratic — f(x)=x2+0.2 sin⁡(5x)f(x)=x^2+0.2\,\sin(5x)🌊 A bowl with ripples. Some areas are smooth, others are bumpy - learning rate needs to be careful!

Tip: try different starts and compare f(xn)f(x_n). Watch the color: blue segments decrease, red indicates an increase (overshoot).

▼
2/L ≈ 1.000
0 < η < 0.5 → smooth convergence, x keeps its sign
decrease overshoot (f increases)
0 steps
Drag the point • Step/Play to apply xn+1 = xn − η f′(xn)
Generic update:xn+1=xn−η f′(xn) \text{Generic update:}\quad x_{n+1} = x_n - \eta\, f'(x_n)

For the quadratic, the closed-form map xn+1=(1−2η)xnx_{n+1}=(1-2\eta)x_n explains the regimes above. For the non-convex models, the widget displays an estimated LL and the no-increase threshold2/L2/L to interpret overshoot.

1D builds intuition.
Next we’ll look at higher dimensions where geometry (conditioning, saddles, narrow valleys) matters even more.

Gradient Descent — 3D Views

We now optimize functions of two variables f: R2 ⁣→ ⁣Rf:\ \mathbb{R}^2\!\to\!\mathbb{R}. Each function f(x,y)f(x,y) is rendered as a 3D surface z=f(x,y)z=f(x,y), so you can see the landscape while gradient descent moves on it. Adjust the start point and the learning rate η\eta, and observe whether the path heads to a minimum—or gets stuck at saddles or in poor basins.

We use three classic landscapes:

  • Quadratic bowl — f(x,y)=x2+y2f(x,y)=x^2+y^2🍜 A perfect bowl shape. No matter where you start, there's only one way down to the center!
  • Himmelblau's function — f(x,y)=(x2+y−11)2+(x+y2−7)2f(x,y)=(x^2+y-11)^2+(x+y^2-7)^2🏔️🏔️🏔️🏔️ Four different valleys! Which one you reach depends on your starting point.
  • Rastrigin (A=10) — f(x,y)=20+[x2−10cos⁡(2πx)]+[y2−10cos⁡(2πy)]f(x,y)=20+[x^2-10\cos(2\pi x)]+[y^2-10\cos(2\pi y)]🧀 Like Swiss cheese - full of holes! Gradient descent often gets trapped in the wrong hole.

Two step rules. Fixed step is plain gradient descent, exactly the formula above: every step is η ∇f\eta\,\nabla f. Line search (backtracking, Armijo) starts from η\eta and halves the step until ff decreases enough: the usual fix when no single η\eta suits the whole landscape.

Tip: try several Random start positions per function and compare final f(x,y)f(x,y). Watch for saddles (gradient ≈ 0 but not a minimum)—descent can stall there.

▼

Plain gradient descent: every step is exactly η·∇f. Too large an η overshoots (red segments). Click on the surface to pick a start point.

Initializing 3D visualization…
Last steps
nxyf(x,y)‖∇f‖
Status
0 steps · f = 36.1750

Gradient Descent in Neural Networks

A neural network adjusts its weights to better predict the right answer. Gradient descent simply means: measure how each weight changes the error, then nudge it a little in the right direction to reduce that error.

1) Predict

🧠 The neural network combines the inputs with its current weights to make a prediction. Like a recipe: mix ingredients (inputs) in certain proportions (weights) to get a result.

2) Measure the error

📏 Compare the prediction with the correct answer. If the network predicted 0.8 but the answer was 0.2, that's a big error!

3) Compute the gradient

🧭 Figure out which weights caused the error. The gradient is like a compass pointing toward "make the error bigger" - so we'll go in the opposite direction.

4) Update weights

⚡ Nudge each weight slightly in the direction that reduces error. The learning rate controls how big these nudges are. Repeat thousands of times → the network gets smarter!

What you’ll see below

The smallest possible network: a single neuron with two inputs (logistic regression), learning to separate two clouds of points.

  • Boundary BEFORE/AFTER: the dashed line (before) moves to the solid line (after) at each update (300 ms animation).
  • Update arrow: inside the neuron, a green arrow shows −η∇-\eta\nabla(direction & step size).
  • Numbers panel: live values of y^, y, e, dW, db, Δw\hat y,\ y,\ e,\ dW,\ db,\ \Delta w for the current sample.
  • Current sample: the highlighted point is the one used for this step (one point at a time: stochastic gradient descent). It turns green while its update is applied.
  • Loss curve: the pale line is the loss on the current sample (noisy, since each step sees a single point); the bold line is the average loss over all points, the quantity gradient descent actually lowers.

Play with η, step manually (Forward → Backprop → Update) or use Auto-play to watch the boundary move.

sample: 1/200|iter: 0|phase: forward|loss: —
Decision boundary — before (dashed) → after (solid)
Backprop visualizer — gradients & update vector
Loss curve (binary cross-entropy)