Back in Module 3 you learned how a network learns: backpropagation nudges the weights to lower the error ("turning the knobs"). That's the engine. But a real chatbot goes through three separate training stages, each using that same engine for a different goal.
Pretraining = a kid reading every book in the library (soaks up knowledge, but blurts random things). Fine-tuning = apprenticeship, watching pros answer questions the right way (learns the job). RLHF = a mentor giving thumbs-up/down until their judgment is polished (learns good taste). Same brain — three phases of growing up.
This is the giant, expensive stage. The model reads a massive slice of the internet — books, articles, code, forums — and does exactly one thing: predict the next word (Module 5!). No labels, no humans grading it. The text itself is the answer key: hide the next word, guess it, check, adjust the weights, repeat trillions of times.
To predict "The capital of France is ___", the model must learn that it's Paris. To finish "Water boils at ___ degrees", it must learn physics. Predicting the next word forces it to absorb facts, grammar, reasoning, and style — all just to guess better. Knowledge is a side-effect of getting good at the guessing game. This is called self-supervised learning (the data labels itself).
The catch: after pretraining, the model is brilliant but wild. Ask it a question and it might just continue with more questions — because on the internet, questions are often followed by more questions. It predicts text; it doesn't yet know it's supposed to help.
Predict the next word to "train" the model. Each correct guess teaches it a fact — watch its knowledge bar fill. This is the whole of pretraining, just tiny.
Now we tame the wild genius. We show it thousands of example conversations written by humans: a question, and an ideal answer. It keeps doing next-word prediction — but now on demonstrations of being helpful. It learns the format: "when I see a question, I give a direct, useful answer" (not more questions).
This is called Supervised Fine-Tuning (SFT) — "supervised" because humans supplied the correct answers. It's a small, cheap stage compared to pretraining (thousands of examples, not trillions of words), but it completely changes the model's behavior.
Fine-tuning doesn't add much new knowledge — the facts came from pretraining. It teaches behavior: how to respond, what tone, when to refuse. Think of it as etiquette school for a genius who never learned how to hold a conversation.
Same question, same knowledge — but see how the model responds before fine-tuning (wild text-predictor) vs after (helpful assistant). Click to fine-tune each example.
The final polish, and the "secret sauce" behind ChatGPT feeling so good. RLHF = Reinforcement Learning from Human Feedback. Fine-tuning taught the model to answer; RLHF teaches it to answer the way people prefer.
The model is a chef cooking dishes. Instead of a human tasting every plate forever (too slow), we train a food critic (reward model) who learned everyone's tastes. Now the chef cooks, the critic scores instantly, and the chef adjusts recipes to score higher — improving fast, without a human at every meal.
RLHF is what makes models helpful, honest, and harmless — it's where "don't produce harmful content" and "admit when unsure" get reinforced. But it's imperfect: the model learns what human raters preferred, which can bake in their biases, or make it "people-pleasing" (agreeing to sound nice). It's alignment, not perfection.
You're the human rater. For each prompt, pick the better answer. Your choices train the "reward model" — and you'll watch the assistant's quality climb as it learns your preferences.
Here's how the three stages connect into the model you talk to every day:
🎉 That's the complete picture — you now understand not just how an LLM is built, but how it's trained to be an assistant. This was the last big gap. You've gone from a single pixel to understanding (and building) the whole modern AI pipeline.
You've mastered the concepts. Now build it all for real, in code. The Applied AI course is the hands-on sequel: train a GPT from scratch in PyTorch, fine-tune with LoRA, and wire up RAG with a vector database — every module pairs real, copyable code with interactive explainers.
⚙️ Continue to the Applied AI course →