RAG gave a model access to documents. But a real assistant also needs to remember you — this conversation, your name, what you asked last week. The catch: an LLM is stateless — it forgets everything between calls. This final module is about the types of memory we bolt on, and exactly when to use each.
The core problem
LLMs are stateless — they forget instantly
Here's the surprising truth: an LLM has no memory of its own. Each API call is independent — it only "remembers" what you put in the prompt this time. The illusion of a continuous chat is created entirely by us re-sending the conversation history every turn.
The illusion of memory
When you chat with ChatGPT, it feels like it remembers — but under the hood, the whole conversation so far is stuffed back into the prompt on every message. "Memory" in AI = deciding what to put back into the prompt, and where to store the rest. That's the entire subject of this module.
The toolkit
Four kinds of memory (and when to use each)
These map neatly onto how human memory works. Click each to explore.
Short-term (context window)
The recent conversation, kept right in the prompt. Fast, exact — but limited by the context window size.
use for: the current chat
Long-term (vector memory)
Facts saved to a vector DB (Module 10) and retrieved by relevance later. Survives across sessions.
use for: "remember my name"
Episodic
Summaries of past sessions/"episodes" — what happened when. Like a diary of interactions.
use for: "last time we discussed…"
Working memory
A scratchpad the model uses during a task — intermediate results, a plan, a running to-do list.
use for: multi-step tasks
👆 Click a memory type above to explore it in depth.
🎮 Game 1
Match AI memory to human memory
Each AI memory type mirrors a human one. Match them to lock in the intuition.
AI memory ↔ human memory
Question 1 of 4
Short-term memory in detail
The context window — and why it fills up
Short-term memory lives in the context window: the max tokens (Module 4) the model can read at once. Every turn, you re-send: system prompt + conversation history + the new message. As the chat grows, history eats the window — until it overflows and you must drop or summarize old messages.
🎮 Game 2
Fill the context window
Add messages and watch the context window fill. When it overflows, the oldest messages get dropped (the model literally forgets them) — unless you summarize. This is the #1 real constraint in chatbots.
System + history + new = the window
Each message costs tokens. Window = 100 (tiny, for demo).
Long-term memory in detail
Long-term memory = RAG on your conversations
Here's the beautiful part: long-term memory is just RAG (Module 11) applied to memories. When something important is said ("my name is Alex", "I'm allergic to peanuts"), save it to a vector DB. Later, retrieve relevant memories by similarity and inject them into the prompt. The model "remembers" — even across sessions, even beyond the context window.
python · long-term memory (= RAG on memories)
# SAVE a memory when something worth remembering happens
memory_db.add(documents=["User's name is Alex, allergic to peanuts"], ids=["m1"])
# RECALL relevant memories before answering (retrieve → augment)
relevant = memory_db.query(query_texts=[user_message], n_results=3)
prompt = f"Known facts about the user:\n{relevant}\n\nUser: {user_message}"
answer = llm.generate(prompt) # now it "remembers" Alex & the allergy
🎮 Game 3
Chat with long-term memory
Walk through a conversation where the assistant saves facts and recalls them later — even after the original message scrolled out of the context window. This is long-term memory in action.
Watch it save & recall a fact
Step through the conversation.
🎮 Game 4
Which memory type fits?
The real design decision. For each need, pick the right memory type. This is exactly what you'd choose when building an assistant.
Pick the right memory for the job
Scenario 1 of 5
🎮 Game 5
Manage a token budget
Real memory design is a budget problem: the context window is finite, so you decide what deserves the space. Drag the sliders to allocate your window across system prompt, recent chat, and retrieved long-term memories — and see if it's a good balance.
Allocate a 4,000-token window
Balance the three claims on your context.
recent chat history: 50%
retrieved long-term memories: 30%
🎮 Game 6
Assemble a full assistant's memory
The finale: a complete assistant uses all four memory types together. Match each part of a single user interaction to the memory type that handles it.
One assistant, four memories working together
Match all 4 · 0/4 done
🎯 Check your understanding
Quick challenge
1. Why do we say an LLM is stateless?
2. Short-term memory in an LLM lives in…
3. Long-term memory is essentially…
4. When the context window overflows, you must…
5. Working memory is best for…
Takeaway
What you learned
LLMs are stateless — "memory" means deciding what to put back in the prompt. Four types: short-term (the context window — recent chat, fast but size-limited), long-term (RAG on saved facts in a vector DB — survives sessions), episodic (summaries of past sessions), and working (a scratchpad during a task). When the window overflows, summarize or drop old turns and push important facts to long-term memory. Real assistants combine all four, budgeting the finite context window across them.
🎓 You finished the Applied AI course!
From tensors to GPT-from-scratch to fine-tuning to RAG and memory — you can now build models AND build real systems with them. Combined with the Fundamentals course, you understand modern AI end-to-end, in theory and in code.
✓ PyTorch & tensors✓ Autograd✓ Training loop✓ Data & tokenizers✓ makemore✓ nanoGPT✓ Fine-tuning✓ LoRA/QLoRA✓ Embeddings search✓ Vector DBs✓ RAG✓ Memory
Where next? Build a real project that combines these — e.g. a RAG chatbot over your own docs with long-term memory, or fine-tune a small model for a specific task. You now have every piece.