What the model remembers

The memory paradox

A LLM has perfect memory — and yet it forgets.

Perfect because everything in its context window is accessible instantly, with absolute precision. The model doesn't "roughly remember" what you said ten exchanges ago: it still sees it, exactly as it was.

Bounded because this window has a maximum size. Beyond it, tokens disappear — permanently, without residue. No gradual degradation, no blur. Just: present or absent.

This explains the strange experience of talking to ChatGPT for an hour on a project, then opening a new conversation and watching it "forget" all that context. That's not a bug — it's a fundamental limitation of the architecture.

What the context window really is

The context window is the number of tokens the model can process in a single pass. Everything that enters this window — your question, the conversation history, system instructions, documents you've pasted — is processed together, in one pass.

The first versions of GPT-3 had windows of 2,048 tokens. Today, a million tokens has become common among frontier models, and Llama 4 Scout has advertised 10 million since April 2025 — with an important caveat: past the million mark, effective recall degrades noticeably. The advertised window is not the usable one. That's a massive evolution, but it doesn't change the nature of the problem: there's always a limit.

And that limit has a direct consequence on cost.

Attention is expensive

Remember the attention mechanism? Each token looks at all the other tokens to compute its weights. If you have n tokens in your window, attention performs n² operations.

Doubling the window doesn't double the cost — it quadruples it.

At 1,000 tokens: 1 million operations.
At 2,000 tokens: 4 million.
At 8,000 tokens: 64 million.
At 128,000 tokens: 16 billion.

That's why providers charge per token. And that's why long conversations cost quadratically more than short ones: when the conversation doubles, the compute quadruples.

Play with the window

Send messages one by one. Watch when old turns fall out of the window — the model can no longer access them, even if you explicitly refer to them.

Change the window size to see how forgetting speeds up or slows down.

Slide the content: anything that scrolls out of the window is forgotten for good. The compute cost of attention grows in O(n²) with context length, which is why providers race for 1M-token windows.

The KV cache, in two words

Computing attention on all tokens at every new step would be catastrophic. The optimization that saves the day is called the KV cache: keep the work already done in memory, and only add the new token's row at each step.

This is what makes real-time generation possible — and what consumes VRAM proportional to context length. We come back to it in detail in chapter 18.

System prompts and chat templates

When you send a message to ChatGPT or Claude, what the model receives isn't exactly your message. It's a carefully formatted sequence that looks something like:

<|im_start|>system
You are a helpful and concise assistant.
<|im_end|>
<|im_start|>user
What's the capital of France?
<|im_end|>
<|im_start|>assistant

This formatting — the chat template — turns a dialogue into a linear sequence of tokens, with special markers distinguishing roles (system, user, assistant). The model has been fine-tuned to recognize them.

The system prompt is a special message placed at the very start. It sets behavior, tone, constraints ("respond in JSON", "never answer questions about topic X", "you are an American IP lawyer"). It lives in the context window like everything else, but its position at the start and its privileged role give it more weight than ordinary messages.

Each family of open models has its own template: ChatML, popularized by OpenAI, [INST]…[/INST] for Llama 2, other variants for Llama 3, Mistral or Qwen. If you run one of these models yourself and type your dialogue by hand without respecting the right format, the model misbehaves — it was trained to see those markers and expects them.

With proprietary APIs (OpenAI, Anthropic, Google), you never see those markers. You send a list of structured messages, each with its role (system, user, assistant), and the formatting is applied server-side. The mechanism is the same; it's just hidden.

Why position in the window matters

A surprising finding, published in 2023 by Liu and colleagues: LLMs don't use all context with equal effectiveness.

Information at the beginning and at the end of the window is better exploited than information in the middle. This phenomenon is called "lost in the middle" — the model tends to ignore passages buried at the center of a long context.

The long-context models of 2025-2026 have greatly mitigated the problem. Finding a specific sentence buried in the middle of a very long document — the so-called needle in a haystack test — is essentially solved today. What still resists is reasoning: as soon as you have to cross-reference several passages far apart from each other, position still matters.

Practical consequence: if you build an assistant with reference documents, put the most important information at the beginning or end of the prompt, not in the middle.

Toward longer windows

Researchers are exploring several approaches to extend windows without exploding costs:

• Sparse attention: instead of looking at all tokens, each token only looks at a subset (nearby ones, or selected by importance).
• Sliding window attention: a sliding window — each token only looks at its k nearest neighbors.
• Flash Attention: not a reduction in computation, but a much more GPU-efficient implementation of exact attention.

But all these techniques have a cost in precision or implementation complexity. There's no perfect solution yet. The inference-side details — and especially how they interact with the KV cache — are in chapter 18.

What this means for you

A few practical rules that follow from all this:

Be concise in your prompts. Every token in context costs memory and compute. A 10,000-token context costs much more than ten times a 1,000-token context.

Structure your documents. If you inject content (notes, articles, code), chunk and filter first — don't paste long blocks where 90% isn't relevant.

Use short sessions. For long iterative tasks, it's often better to start a new conversation with a summary of the context rather than letting a giant conversation grow.

And if you need to give the model access to very large amounts of information — thousands of pages, a database — the context window is not the right solution. That's where the next chapter comes in.

Updated August 2, 2026