Give the model thousands of real names — emma, olivia, liam, noah… — and it learns the patterns of how names are spelled. Then it generates brand-new ones that sound like names but never existed: emmalyn, korvin, adalie.
The trick: a name is just a sequence of characters. If the model can predict "what letter comes next?" — one character at a time — it can build whole names. That's the exact same job as predicting the next word (fundamentals Module 5), just at the letter level.
Train on e→m→m→a→[end] and thousands more. To generate: start with the "beginning" marker, predict the next char, append it, feed it back, repeat until the "end" marker. It's auto-regression (fundamentals Module 12) at the character level.
First, map every character to a number (Module 4's tokenizing, but per-character). We add a special . token to mark the start/end of a name. Click a letter to see its ID.
chars = "." + "abcdefghijklmnopqrstuvwxyz" # 27 tokens ('.' = start/end)
stoi = {ch:i for i,ch in enumerate(chars)} # string → int
itos = {i:ch for ch,i in stoi.items()} # int → string
print(stoi["e"]) # 5The simplest possible name model: for every pair of characters, count how often B follows A in the training names. That gives a 27×27 table of counts. Turn each row into probabilities, and you can already generate names (badly). This is fundamentals Module 5's bigram — now with real matrices.
import torch
names = """emma olivia ava isabella sophia mia charlotte amelia harper evelyn
abigail emily ella elizabeth camila luna sofia avery mila aria scarlett
penelope layla chloe victoria madison eleanor grace nora riley zoey hannah
hazel lily ellie violet lillian zoe stella aurora natalie emilia everly
leah aubrey willow addison lucy audrey bella nova brooklyn paisley savannah
claire skylar isla genesis naomi elena caroline eliana anna maya valentina
ruby kennedy ivy ariana aaliyah cora madelyn alice kinsley hailey gabriella
allison gianna serene samantha sarah autumn quinn eva piper sophie sadie
liam noah oliver elijah james william benjamin lucas henry alexander mason
michael ethan daniel jacob logan jackson levi sebastian mateo jack owen
theodore aiden samuel joseph john david wyatt matthew luke asher carter
julian grayson leo jayden gabriel isaac lincoln anthony hudson dylan ezra
thomas charles christopher jaxon maverick josiah isaiah andrew elias joshua
nathan caleb ryan adrian miles eli nolan christian aaron cameron ezekiel
colton luca landon hunter jonathan santiago axel easton cooper jeremiah
angel roman connor jameson robert greyson jordan ian carson jaxson leonardo
nicholas dominic austin everett brooks xavier kai jose parker adam jace
wesley kayden silas bennett declan waylon weston evan emmett micah ryder""".split()
chars = ".abcdefghijklmnopqrstuvwxyz"
stoi = {ch:i for i, ch in enumerate(chars)}
itos = {i:ch for ch, i in stoi.items()}
N = torch.zeros((27, 27), dtype=torch.int32) # counts table
for name in names:
chs = ["."] + list(name) + ["."] # .emma.
for a, b in zip(chs, chs[1:]): # each adjacent pair
N[stoi[a], stoi[b]] += 1 # tally it
P = N.float()
P = P / P.sum(1, keepdim=True) # rows → probabilities
print("Count of e → m:", N[stoi["e"], stoi["m"]].item())
print("Every probability row sums to:", P[stoi["e"]].sum().item())Here's the 27×27 table (brighter = "this letter often follows that one"). Hover/click a cell to read the pair. Notice the bright column at . — lots of names end in certain letters.
This is sampling: start at ., look at the probabilities for the next letter, roll a weighted die, append the winner, repeat until you hit . again. Step through it and watch a name appear.
# Run the "counting bigrams" cell above first to create P and itos.
g = torch.Generator().manual_seed(42) # reproducible sampling
ix = 0 # start at '.'
out = []
while True:
p = P[ix] # next-char probabilities
ix = torch.multinomial(p, num_samples=1, generator=g).item()
if ix == 0: break # hit '.' → name is done
out.append(itos[ix])
print("".join(out)) # a new name!Run the sampler many times to spit out fresh names. Some sound great, some are gibberish — that's the bigram's limit (it only looks back one letter). We'll fix that with a neural net.
def generate_name(temperature=0.75):
ix, out = 0, []
while len(out) < 12:
p = P[ix].pow(1.0 / temperature) # favor likely transitions
p = p / p.sum()
ix = torch.multinomial(p, 1).item()
if ix == 0:
break
out.append(itos[ix])
return "".join(out)
generated = []
while len(generated) < 8:
name = generate_name()
if (3 <= len(name) <= 10
and name not in names
and name not in generated):
generated.append(name)
print(generated)Counting works but is rigid. The neural version learns those same probabilities with a trained weight matrix — using the exact training loop from Module 3. Instead of tallying, it predicts and improves via gradient descent. Same input/output, but now it can grow smarter (more context, more layers).
import urllib.request
import torch
import torch.nn.functional as F
# 1. Load the real makemore name dataset.
url = "https://raw.githubusercontent.com/karpathy/makemore/master/names.txt"
names = urllib.request.urlopen(url).read().decode("utf-8").splitlines()
# 2. Map characters to IDs: '.' is both start and end.
chars = ".abcdefghijklmnopqrstuvwxyz"
stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for ch, i in stoi.items()}
# 3. Build every (current character, next character) training pair.
xs, ys = [], []
for name in names:
sequence = ["."] + list(name) + ["."]
for current, next_char in zip(sequence, sequence[1:]):
xs.append(stoi[current])
ys.append(stoi[next_char])
xs = torch.tensor(xs)
ys = torch.tensor(ys)
# 4. Train a 27 × 27 neural bigram weight matrix.
g = torch.Generator().manual_seed(2147483647)
W = torch.randn((27, 27), generator=g, requires_grad=True)
for step in range(100):
xenc = F.one_hot(xs, num_classes=27).float()
logits = xenc @ W
loss = F.cross_entropy(logits, ys)
W.grad = None
loss.backward()
with torch.no_grad():
W -= 50 * W.grad
if step % 10 == 0 or step == 99:
print(f"step {step:2d} | loss {loss.item():.4f}")Run this cell after the training cell. It samples directly from the learned neural weights without spelling filters, so you see the bigram model's real quality.
# Requires W, stoi, itos, and names from the training cell above.
# No fixed seed: running this cell again produces a fresh random batch.
# To reproduce the same batch, uncomment: torch.manual_seed(42)
def generate_neural_name(temperature=1.0, max_length=20):
ix = stoi["."] # begin at the start token
output = []
for _ in range(max_length):
x = F.one_hot(torch.tensor([ix]), num_classes=27).float()
with torch.no_grad():
logits = x @ W # learned next-character scores
logits = logits / temperature
probabilities = F.softmax(logits, dim=1)
ix = torch.multinomial(
probabilities[0],
num_samples=1,
).item()
if ix == stoi["."]: # end token
break
output.append(itos[ix])
return "".join(output)
# Generate raw samples and check whether they appeared in training.
training_names = set(names)
samples = [generate_neural_name(temperature=1.0) for _ in range(20)]
for i, name in enumerate(samples, start=1):
status = "SEEN" if name in training_names else "NEW"
print(f"{i:2d}. {name or '[empty]':20s} {status}")
nonempty = [name for name in samples if name]
novel = [name for name in nonempty if name not in training_names]
average_length = sum(map(len, nonempty)) / len(nonempty) if nonempty else 0
print("\nFinal training loss:", round(loss.item(), 4))
print("Non-empty names:", len(nonempty), "/", len(samples))
print("Novel names:", len(novel), "/", len(nonempty))
print("Average length:", round(average_length, 2))Loss measures next-character prediction. NEW checks novelty, not name quality. Read the raw samples yourself: a bigram only remembers one character, so some new outputs will still be awkward. Temperature below 1.0 is safer and more repetitive; above 1.0 is more varied and chaotic.
softmax (Module 3), backward (Module 2), the step (Module 2), cross-entropy loss (Module 3), one-hot inputs (tokens, Module 4). makemore is just those pieces assembled. The neural bigram lands at about the same quality as counting — but now the architecture can grow.
Press train and watch the loss drop as the network learns the same letter patterns the counts had, but by gradient descent this time. The page performs the real bigram weight updates on its training-name subset.
names.txt, so its exact loss floor will differ.makemore's whole arc is climbing this ladder: each rung sees more context and produces better names (lower loss). Click each to see what changes. Rung by rung, this leads straight to a Transformer (next module!).
. token for?torch.multinomial(p, 1) do when sampling?. start/end token), then predict the next character repeatedly and sample until . — that's auto-regression at the letter level. The bigram (count / neural) looks back one char; you generated real names from it. The neural version reuses everything: softmax, cross-entropy, backward, gradient step. Climbing the ladder (bigram → MLP → more context) gives better names and lower loss — and leads straight to the Transformer.Next up → Module 6: nanoGPT ⭐ — the big one. We take this exact idea and build a real GPT from scratch: tokens, embeddings, self-attention, transformer blocks, and generation. Every fundamentals concept, as working code.