Last module, the RNN's "notebook" (hidden state) let it remember "river" to disambiguate "bank." That works for a short sentence. But your notes name the catch precisely: as the sentence gets longer, the "ink" in the notebook fades. By the end of a long paragraph, the RNN has often forgotten what happened at the very beginning.
Remember the canvas from Module 7? You throw blue paint for "River," then 49 more layers of paint for the rest of a 50-word sentence. The original blue is buried — invisible. The reason it fades isn't random; it's a specific mathematical effect called the vanishing gradient. Let's unpack it.
Recall training (Module 3): the network learns by sending the error signal backward to nudge each weight — that backward signal is the gradient. In an RNN, to learn a connection between word #1 and word #50, that signal has to travel back through all 50 steps.
Here's the killer: at each step backward, the signal gets multiplied by a small number (less than 1). Multiply by a fraction 50 times, and the signal shrinks toward zero. By the time it reaches word #1, there's essentially nothing left to learn from. The connection is never learned.
The gradient across many steps is a product of many small factors:
Each factor is small because of how derivatives of squashing functions (like tanh/sigmoid) behave. Many small numbers multiplied together race to zero.
Set the "per-step factor" (how much the signal survives each step) and the number of steps. Watch the gradient that reaches the first word. Try 0.6 over 25 steps — it's basically zero. This is exactly why distant words can't be learned.
Notice: even a factor of 0.9 dies over 50 steps. Only a factor of exactly 1.0 preserves the signal — and that's the trick LSTMs use.
You know this game: whisper a message down a line of people, and by the end it's garbled nonsense. Each person loses a little. That's exactly the vanishing gradient — each step (person) degrades the signal until the original message is lost.
A photocopy of a photocopy of a photocopy… — each copy is slightly fainter, until the text is unreadable. An echo in a canyon — each bounce is quieter until you hear nothing. Chinese whispers. In all of them, a signal passing through many stages, each losing a bit, ends up gone. That's the vanishing gradient.
Slide the "gap" between the key clue and the word that needs it. Watch the RNN's memory of that clue drain as the gap grows. Short gap = remembered. Long gap = forgotten.
The immediate solution (from your notes) was the LSTM — Long Short-Term Memory. It's a smarter RNN with gates that decide what to keep, what to forget, and what to output. Crucially, it has a "memory highway" (the cell state) where important information can flow forward unchanged — factor ≈ 1.0 — so the ink doesn't fade as fast.
A vanilla RNN scribbles everything into one messy notebook that gets overwritten. An LSTM adds a conveyor belt running alongside: it can place important notes ("Subject: Dog") on the belt and they ride forward untouched, while unimportant stuff is dropped. Gates are the hands that decide what goes on the belt and what falls off.
LSTMs stretched memory from ~20 words to a few hundred — a big win. But they're still sequential (they read word-by-word, so they're slow) and still eventually forget across very long documents. The real breakthrough wasn't a better memory — it was letting the model look at ALL words at once. That idea is attention, and it changed everything. Next module.
🎉 Part C's setup is complete — you now understand the exact wall that RNNs hit. Next → Module 9: Attention — the big idea ★. The paper that changed AI ("Attention Is All You Need") threw out sequential memory entirely and let every word look at every other word, directly. This is the turning point of the whole course.