Tokens, Context & Memory
Why can AI handle long documents but occasionally lose track of something said earlier? These three ideas explain a lot.
Tokens
Text is processed in pieces called tokens. One token may be a word, part of a word, or punctuation. Tokens are not the same as words.
Context window
The amount of information the model can consider at one time. Very long conversations or files may exceed this limit.
Memory
Some AI products can retain or retrieve useful information across chats. This is different from the text currently in the conversation.
A simple analogy
Tokens: Pieces of the material. Context: What's on the desk right now. Memory: Notes saved or retrieved for later. The exact behavior depends on the product and settings.
Live demo: A conversation with limits
- Ask AI to summarize three short paragraphs you provide.
- Ask it to use one detail from the paragraphs in a new email.
- Discuss what could happen if the conversation becomes very long or the earlier text is no longer in context.
Quick exercise
Ask: “Explain tokens and context windows as if you're teaching a 12-year-old, using a backpack analogy. Keep it to four sentences.” Then ask one follow-up question.
Do not teach a fixed tokens-per-word rule. Tokenization varies by language and content. Do not promise that the product remembers everything.