One of the earliest language models was the bigram. The idea is beautifully simple: given one word, predict the word most likely to come next. Chain those predictions and you generate a whole sentence — one word at a time.
You've used a bigram thousands of times: the next-word suggestions above your phone keyboard. Type "good" and it offers "morning / luck / night." It's just guessing the most common word to follow the last one you typed. Tap-tap-tap and it writes a sentence for you — often a nonsensical one. That's a bigram.
You're the bigram. Start with a word, then keep picking the next word from the model's top suggestions (with their probabilities). Watch your sentence grow — and notice how quickly it drifts into nonsense, because each choice only looks at the single previous word.
The earliest bigram, Maximum Likelihood Estimation (MLE), isn't clever at all — it just counts. It scans a big pile of text and tallies: "after the word cat, how often does each word appear?" Those counts become probabilities.
Two ways to pick: greedy decoding always takes the single highest-probability word (deterministic, but can get stuck in loops), or sample randomly weighted by the probabilities (adds variety — this is temperature from Module 3!).
Here's a tiny training text. Click a word to highlight every place it appears, and the model counts what came right after it — turning raw counts into next-word probabilities, exactly like MLE.
Bigrams feel magical for two words and fall apart for two sentences. Here's why — and these failures are exactly what pushed the field toward everything that came after.
A bigram only ever sees the single previous word. It has zero context beyond that. So it forgets the subject, the topic, everything older — which is why chained bigram text wanders into gibberish.
"I do not like this" and "I do like this" mean the opposite — but a bigram treats "not" as just another word to count. It has no understanding, only statistics. Try the demo below.
Two sentences that mean opposite things. A meaning-aware model sees the flip; a bigram, only counting word-pairs, treats them almost the same. Press the button and compare.
Greedy decoding always grabs the most likely word. But if "mat" is most likely after "the," and "the" is most likely after "mat"… you get an infinite loop. Press Generate (greedy) and watch it get stuck — then try Generate (random) to escape.
This "greedy loop" is literally the problem in your notes that pushed models from bigrams → trigrams (look at the last two words) — and eventually to models that look at everything.
The obvious fix for "no context past one word" is: look at two words instead of one. That's a trigram — it predicts the next word from the previous pair. This small change fixes a surprising amount: it catches "not like" as a unit, and it breaks many greedy loops (because "the mat" and "mat the" lead to different places).
bigram = last 1 word → next. trigram = last 2 words → next. 4-gram = last 3, and so on. The general name is an n-gram: use the previous n−1 words to predict word n. More context = better predictions… but also exponentially more combinations to store.
If a friend just says "bank", you have no idea what they mean (money? river?). But "river bank" — now you know. A bigram only heard the last word; a trigram heard a bit more, so it guesses better. It's the difference between catching one word of a conversation vs. catching the last few.
Same sentence, two models. Flip between bigram (looks at 1 word) and trigram (looks at 2 words) and watch how much of the sentence each one "sees" when predicting the next word — and how the prediction improves.
Notice: with the phrase "do not", the trigram's two-word window captures the "not" together with the verb — so it can flip the meaning. The bigram, seeing only one word, still can't.
Remember Game 4, where the bigram got stuck in the → mat → the → mat…? A trigram remembers the pair it's in, so "the mat" and "mat the" predict different next words — and the loop breaks. Compare them side by side.
Since words are vectors, we can replace the counting with a small neural network that predicts the next word. This is a real upgrade over pure counting:
Instead of a lookup table of counts, the neural version runs the word through a tiny network: embed → hidden layer → softmax → next-word probabilities. It's the exact machine from Modules 1–3, now applied to words.
Pick an input word and press Predict. Watch it become an embedding (Module 4), flow through a hidden layer (Module 2's neurons), and come out as softmax probabilities (Module 3) over the next word. Everything you've learned, working together.
Here's what counting can't do. The training text contains "the dog ran" but never "the cat ran". A counting bigram gives "cat ran" a probability of zero. But because the neural model knows dog ≈ cat (similar embeddings from Module 4), it predicts "ran" anyway. Test it.
Even a neural bigram still only looks at the last word. Trigrams stretch this to the last two words — better, but still tiny. The real question becomes: how do we give a model memory of everything that came before? That question is the whole rest of this course — RNNs (Module 6), then attention (Module 7).
Next up → Module 6: Reading the mood (sentiment analysis) — a gentle, tangible NLP task before we tackle the big stuff. Then Part C: memory (RNNs) and the attention breakthrough.