A bigram sees the last word. A trigram sees the last two. But language depends on words that appeared far earlier. Consider the classic puzzle from your notes:
The same word "bank" means different things — and the clue is the word "river", which came five words earlier. A bigram never sees it. We need a model that remembers.
An RNN adds one powerful thing: a hidden state. It's a vector that acts as a running memory. As the model reads each word, it (1) looks at the new word and the current hidden state, (2) produces an output, and (3) updates the hidden state to carry forward. That updated memory feeds into the next word's step — that's the "recurrent" loop.
Imagine reading a mystery novel. A bigram is a reader who only remembers the very last word — sees "dead," guesses "body," but has no clue who died. An RNN carries a Summary Notebook (the hidden state). Every new word, it glances at the notebook, jots an update, and passes it on. So the memory of page 1 still shapes how it reads page 50.
At each step t, the new hidden state combines the previous state and the current word:
hₜ = new memory · hₜ₋₁ = old memory · wordₜ = current word. The same weights W, U are reused at every step — that's what makes it "recurrent." (The tanh squashing function — we'll do a deep-dive on activations at the end of the course.)
Step through a sentence and watch the RNN's hidden-state "notebook" update at each word. The memory of "river" stays on the page — so when we reach "bank," the model still remembers it's a river bank.
The same sentence ending, two different earlier clues. See how a bigram (last word only) guesses the same thing for both, while the RNN uses its memory of "river" vs "money" to guess correctly.
Your notes use a brilliant metaphor: the hidden state is a canvas. Each word throws a splash of its color onto it. Early on, colors are vivid. But throw 50 words of paint, and the first color ("river," in blue) gets buried — the model can't see it anymore. Add words and watch "river" disappear.
When the early "paint" gets buried, the model literally can't learn from it — the signal shrinks toward zero as it flows back through many steps. This is the Vanishing Gradient Problem, and it's so important it gets its own module next. It's the flaw that eventually forced the invention of attention.
From your notes: "The dog chased the cat because it was hungry." Who's hungry — the dog or the cat? A bigram has no idea (it forgot "dog" long ago). An RNN kept "Subject: Dog" in its notebook. Press play to see the RNN link "it" back to the right word.
RNNs were a huge leap, but that canvas fades: they have bad long-term memory. On a 50-word paragraph, the beginning is forgotten. This led to LSTMs (Long Short-Term Memory) to keep the "ink" fresh longer — and ultimately to attention. That's the next stretch of the course.
hₜ = tanh(W·word + U·hₜ₋₁ + b)). Like a Summary Notebook, it lets the start of a sentence shape the end, so "river bank" ≠ "money bank" and "it" can point to the right subject. But the memory is a canvas that gets painted over: on long text the early words fade — the vanishing gradient problem, which is next.Next up → Module 8: The Vanishing Gradient Problem — a proper look at why that canvas fades, why it capped RNNs, and why it forced the field to invent something radically new: attention.