A model's error (loss) depends on its weights. Picture the loss as a hilly landscape — high where the model is wrong, low where it's right. Training means walking downhill to the lowest point.
To walk downhill you need to know which way is down — the slope. In math, the slope of the loss with respect to a weight is called the gradient. Autograd = "automatic gradients": PyTorch computes every slope for you, automatically, no calculus by hand.
You're on a hill in thick fog, trying to reach the valley. You can't see far — but you can feel the slope under your feet. Step in the steepest-downhill direction, repeat. That's gradient descent. The gradient is "which way is downhill and how steep," and autograd is the sense of touch that measures it.
Here's a simple loss curve loss = x². Drag x and watch the gradient (the slope) change. Notice: the sign of the gradient always points away from the bottom — so stepping against it walks you down.
import torch
x = torch.tensor(2.0, requires_grad=True) # track gradients for x
loss = x ** 2
loss.backward() # compute the gradient
print(x.grad) # tensor(4.) — matches 2x = 2*2 = 4 ✓Autograd in PyTorch is just three things working together. You already saw them above:
# 1. requires_grad=True → "watch this value, I'll want its slope"
w = torch.tensor(3.0, requires_grad=True)
# 2. do some math (PyTorch secretly records every step)
loss = (w - 5) ** 2 # we "want" w to be 5
# 3. .backward() → walk the recorded steps backward, filling in slopes
loss.backward()
print(w.grad) # tensor(-4.) → negative slope → increase w to lower loss.backward() actually doesAs you do math on tensors, PyTorch quietly builds a computation graph (a record of every operation). Calling .backward() walks that graph backward — applying the chain rule from calculus — to compute how much each input affected the output. That number lands in .grad. This IS backpropagation from fundamentals Module 3, done automatically.
PyTorch builds a graph of your math (forward), then .backward() flows gradients back through it. Step through both passes for loss = (a·b + c)².
Great question, and it's the whole point of backprop. Backward asks one thing at every node: "if I nudge this value up a tiny bit, how much does the final loss change?" That number is the gradient. We work right → left, and every node only needs two things: its own operation, and the gradient arriving from the right.
loss = e², its derivative is 2·e. With e = 7: 2 × 7 = 14.+ operation passes the gradient straight through, unchanged, to both inputs. (Nudge either input of an addition, and the sum changes by the same amount.)d = a·b, each input's gradient is multiplied by the other input's value. Incoming grad is 14, so a.grad = 14 × b = 14 × 2 = 28.b.grad = 14 × a = 14 × 3 = 42. (b gets a's value, a gets b's value.)That's the entire trick — just two local rules applied over and over:
➕ Addition → passes the gradient through unchanged to both inputs.
✖️ Multiplication → each input's gradient = incoming grad × the other input's value.
Chain these together right-to-left and gradients flow all the way back — automatically. That's exactly what .backward() does.
a.grad = (loss→e) × (e→d) × (d→a) = 2e × 1 × b = 14 × 1 × 2 = 28 ✓ — matches!
That toy graph is abstract. Here's the same idea in a real LLM training on one example — predicting the next word. It also clears up the most common confusion: the gradient is not a score you lower — it's a "fix-it signal" you follow.
Training sentence: "The cat sat on the ___" · correct next word: "mat". Watch the model improve over training:
You do not "lower the gradient to lower the loss." The cause-and-effect is:
follow the gradient → update weights → the LOSS drops (that's the goal)
The gradient shrinking toward ~0 is a symptom, not the goal — as the model gets things right, there's less to fix, so the fix-it signal naturally fades (flat ground = tiny slope). Watch the loss, not the gradient.
When people train real LLMs, they literally stare at one thing — the loss curve going down. That downward slope is "learning nicely":
Wrong guess → high loss + loud gradient → big weight change. As it learns → loss drops (the win) and the gradient quiets to ~0 on its own (nothing left to fix). You'll watch exactly these loss curves fall when you build makemore and nanoGPT later!
Now use the gradient to actually learn. Each "step" nudges x against its gradient. Press step repeatedly (or auto-run) and watch the ball roll to the minimum. This is the entire training loop in miniature.
x = torch.tensor(2.5, requires_grad=True)
lr = 0.1 # learning rate = step size
for step in range(20):
loss = x ** 2 # our simple loss
loss.backward() # get gradient into x.grad
with torch.no_grad(): # update without tracking
x -= lr * x.grad # step downhill
x.grad.zero_() # reset grad for next step (important!)The learning rate is how big each downhill step is. Slide it and watch what happens: too small = crawls forever; too big = overshoots and explodes; just right = smooth descent. (We'll go deeper on this in Module 3.)
Test your intuition. For each situation, which way should the weight move to lower the loss? (Remember: step against the gradient.)
Autograd isn't magic — it's ~100 lines (that's Karpathy's "micrograd"). Reveal the logic step by step to see exactly what PyTorch does under the hood for d = a*b; e = d+c; loss = e².
PyTorch accumulates gradients — every .backward() adds to .grad. If you forget x.grad.zero_() (or optimizer.zero_grad()) between steps, gradients pile up and training breaks. It's the #1 beginner bug. You'll see the clean version in Module 3.
requires_grad=True do?.backward() compute?.zero_() / zero_grad() each step?requires_grad=True, do your math (it records a computation graph), call .backward() to fill every .grad via the chain rule (= backpropagation), then step: x = x − lr × grad. Reset grads each step with .zero_(). That loop, repeated, is how every model learns.Next up → Module 3: The Training Loop — we assemble autograd into a real, complete for epoch in ... loop: loss functions, optimizers (SGD, Adam), learning rate scheduling, and how to spot overfitting with a train/validation split.