The network's weights and biases start as random numbers, so its first guesses are nonsense. Training fixes this with a repeating loop: show it an example, check how wrong it was, and nudge every weight slightly toward being less wrong. Do this millions of times, and it gets good.
Picture the network as a giant sound mixing board with thousands of knobs (each weight is a knob). At first they're all set randomly and the sound is noise. Training tells you which way to turn each knob, and how much, to make the music clearer. Turn them a tiny bit, listen, turn again — repeat until it sounds right.
To improve, the network first needs to measure its mistakes. When an input passes through, we compare the output to the correct label. The difference is the cost: bigger error = higher cost. A perfect answer has cost near 0.
Since the cost depends entirely on the weights and biases, it can be written as a cost function. A simple one is the squared error — square each mistake and sum them:
The cost function is the network's compass: it tells training which direction reduces error. Training's whole goal is to make this one number as small as possible.
You're throwing darts at a bullseye. The cost is simply how far your dart landed from the center. A dart in the bullseye = cost 0. A dart on the wall = huge cost. You don't need to know why it missed to know that a dart 30 cm away is worse than one 5 cm away. Cost is just that distance-from-perfect, turned into a single number. And notice we square it (in the formula): missing by 10 is punished four times as hard as missing by 5 — big blunders hurt way more than small ones, so the network fixes them first.
This tiny network should output [0, 1] (meaning "this is a 1, not a 0"). Its 3 weights start random, so the cost is high. Drag the knobs to make the prediction match the target and drive the cost toward 0. This is exactly what backpropagation does automatically — you're doing it by hand.
Minimizing the cost is like adjusting a shower to the perfect temperature. Too cold, you turn it hotter; too hot, you turn it back a bit. Each small turn, you feel the result and adjust again — until it's just right (cost = 0). You're not solving an equation; you're nudging a knob, checking, and nudging again. That's exactly what you're doing with the weights above — and exactly what training automates.
You tuned 3 knobs by hand. A real network has millions — you can't guess. Backpropagation is the algorithm that calculates, for every single weight, which direction and how much to nudge it to reduce the cost. It works backward from the output error, layer by layer, assigning blame to each weight.
Imagine the cost as a hilly landscape and the network standing on a slope. Backprop computes the downhill direction at its current spot; the network takes a small step that way. Repeat, and it rolls toward the valley — the point of lowest cost. The step size is called the learning rate: too big and it overshoots, too small and it crawls.
A restaurant serves a dish and the customer says "too salty." Backpropagation is tracing that complaint backwards through the kitchen to assign blame: the plater added a pinch, the cook added a spoon, the prep station over-salted the stock. Each person gets told exactly how much they contributed to the saltiness — so each adjusts by the right amount, not randomly. The final error (too salty) is passed backward, step by step, and everyone who touched the dish gets a precise correction. That's why it's called back-propagation: the error propagates back to every weight that caused it.
One full pass over the entire training dataset is called an epoch. Networks train for many epochs, the cost dropping each time — until it plateaus at "good enough."
Press Train and watch the automatic version: the cost curve descends epoch by epoch as backprop tunes the weights, and accuracy climbs. This is the graph you stare at when training any real model.
The training curve is your progress learning to ride a bike. Day 1 you fall constantly (cost is high). Each day you fall a little less — the curve drops fast at first because you're fixing the big, obvious mistakes. Then it flattens out: once you can ride, you're only shaving off tiny wobbles, so improvement slows. That steep-then-flat shape is every learning curve — cramming for an exam, a new video game, a language. Big gains early, diminishing returns later.
The 10 output neurons produce raw scores. To read them as probabilities (percentages that sum to 100%), we apply softmax. And a knob called temperature controls how confident vs. exploratory those probabilities are — this same knob controls creativity in LLMs later.
Low temp → the top choice dominates (near 100%). High temp → probabilities flatten, giving underdogs a chance. In an LLM: low temp = focused/repetitive, high temp = creative/random.
Low temperature = a decisive person: "I always get the cappuccino." 99% cappuccino, every time — reliable but boring. High temperature = an adventurous person: "Surprise me!" — today a matcha, tomorrow a mango soda. Same menu, same preferences underneath — temperature just decides whether you play it safe or take a gamble. That's why LLMs feel repetitive at low temp and wild/creative at high temp.
🎉 That completes Part A — Neural Networks. You now know how a network is built and how it learns. Next → Part B, Module 4: Words as Numbers — how do we turn language into vectors? That's where embeddings and the famous "King − Man + Woman = Queen" come in.