Datasets & Tokenizers

What a dataset really is

A dataset is just a big table of examples: inputs (x) paired with correct answers (y). The model studies these pairs to learn the pattern. That's it — no magic.

from datasets import load_dataset

# download a real sentiment dataset in one line
data = load_dataset("imdb")
print(data)                       # train / test splits
print(data["train"][0])          # {'text': '...', 'label': 1}

Explore a dataset

Click through examples like you would after loading one. Real datasets are just rows of x → y. (Simulated preview of what load_dataset gives you.)

Tokenizers — turning text into numbers

A model can't read letters — only numbers. A tokenizer does two jobs: (1) split text into chunks called tokens, and (2) map each token to a number (its ID) from a fixed vocabulary. (You met this conceptually in fundamentals Module 12 — here it's the real tool.)

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")

text = "Today is a mesmerizing day with lot's of memories"
ids = tok(text)["input_ids"]
tokens = tok.convert_ids_to_tokens(ids)

print(ids)
print(tokens)
print(list(zip(ids, tokens)))  # pair every ID with its token
print(tok.decode(ids))

Tokenize your own text (live)

Type anything and watch it split into tokens with IDs — the way a real tokenizer would. Try a long/rare word and see it break into sub-word pieces.

Tokens ≠ words — the counter

This matters for real reasons: API costs and context limits are counted in tokens, not words. Slide the word count and see the token estimate (~1.33 tokens per word).

Split the data (train / val / test)

From Module 3 you know why to split. Here's how — slide the split and see the proportions. The standard is roughly 80 / 10 / 10.

# split a dataset 80/20 (then split the 20 into val/test)
split = data["train"].train_test_split(test_size=0.2)
train = split["train"]        # 80%
temp  = split["test"].train_test_split(test_size=0.5)
val, test = temp["train"], temp["test"]   # 10% / 10%

Batching — feed data in chunks

You don't feed one example at a time (too slow) or all at once (too much memory). You feed batches — small groups. The batch size is how many examples per training step. PyTorch's DataLoader handles this.

from torch.utils.data import DataLoader

loader = DataLoader(train, batch_size=32, shuffle=True)

for batch in loader:      # the loop from Module 3 iterates batches
    pred = model(batch["x"])
    ...                    # forward, loss, backward, step

Order the data pipeline

Put the steps of a real data pipeline in the right order. Click them in sequence — from raw text to a batch the model can train on.

Follow 12 examples through the whole pipeline

Here is the full journey from raw rows to model updates. The numbers stay visible so you can see what changes at each stage.

From token IDs to 768-number vectors

The tokenizer stops at integer IDs. GPT-2's embedding layer uses each ID as a row number in its learned lookup table and returns one 768-number vector per token.

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModel.from_pretrained("gpt2")

text = "Today is sunny"
inputs = tokenizer(text, return_tensors="pt")
ids = inputs["input_ids"]
tokens = tokenizer.convert_ids_to_tokens(ids[0])

# Each ID selects one row from GPT-2's embedding table.
embedding_layer = model.get_input_embeddings()
vectors = embedding_layer(ids)

print("Tokens:", tokens)
print("IDs:", ids)
print("Vector tensor shape:", vectors.shape)
# torch.Size([1, 3, 768])

for token, token_id, vector in zip(tokens, ids[0], vectors[0]):
    print(f"\n{token!r} (ID {token_id.item()})")
    print("First 10 of 768 numbers:", vector[:10])

# ID 8888 directly retrieves the same vector as "Today".
today_vector = embedding_layer.weight[8888]
print("\nToday vector shape:", today_vector.shape)
# torch.Size([768])

Quick challenge

What you learned

Next up → Module 5: makemore — a name generator. Now you have all the pieces (tensors, autograd, training loop, data). We combine them to build a real character-level model from scratch that invents new names — the concepts from fundamentals Modules 5–8, in working PyTorch.