DoNotEnergize · Episode 02

What Is a Token?

Your little question may have brought luggage. Step into the arcade for tokens, context, and limits.

The film

3:35 · AI-assisted educational film

The pieces on the worktable

For text, tokens are the pieces into which a tokenizer divides the material. A token may contain a word, part of a word, punctuation, or spaces. The exact split depends on the tokenizer.

Input can include instructions, conversation history, and documents supplied to the model. Generated text uses output tokens. Service limits and charges vary by model and provider.

A context window limits how much material a model can work with in a request. A rate limit concerns how much usage is allowed over a period of time. They are different limits.

Adding more material does not guarantee a better answer. Choose the information that helps with the actual task.

These are companion notes, not a verbatim transcript.

Further reading: What are tokens? (opens in a new tab)

Full spoken transcript

The words of the published lesson. The key ideas above remain companion notes.

You ask AI one little question. So why does someone suddenly say, we're spending too many tokens? That sounds as though the AI escaped into an arcade. It did not. When an AI model works with text, it processes small pieces called tokens. A token might be a whole word, part of a word, or punctuation. Spaces can be included in those pieces too. The exact split depends on the model. So counting words and counting tokens are not quite the same thing. Input tokens, what goes in. Output tokens, what comes back. But the input can include more than the question you just typed. Instructions, conversation history, documents, information fetched from other systems. Your little question may have brought luggage. All of that gives the model material to work with, which brings us to the work table. The model’s context window is the amount of material it can work with at once, including room for its answer. Imagine asking someone to find the holiday policy. You could hand them the relevant document, or you could empty the entire filing cabinet onto their desk and say, it's probably in there. You have certainly given them more information. You have not necessarily given them more help. The same principle applies here. Relevant context helps. Unnecessary context can bury the thing that matters. And that work table has edges. A context limit tells you how much fits at once. A rate limit tells you how many requests, or how many tokens, a service allows over a period of time. The size of the table, the pace of the work, different limits. Tokens also matter because some AI services charge according to how many they process. What goes in and what comes back can have different prices. But a token count is not an intelligence score. And information sitting on the work table is not a promise that the system will remember it next week. So before you paste 80 pages into a prompt, ask, what am I asking it to do? Which information does it need? And how much answer would actually help? Sometimes it needs all 80 pages. Sometimes it needs paragraph three. More is not the goal. Enough of the right information is. Understand first, automate second. This is DoNotEnergize. Next time, what does autonomous actually mean? Spoiler, it does not mean haunted.