Context window, memory, and why agents forget

In 3 linesCopy Anchor Link

• What it is: The context window is all the text a model can see at once while it writes an answer. It is the model's short-term memory.
• Why it matters: The model keeps nothing between calls. Anything not in the window, the agent has simply "forgotten."
• Example: A support agent forgets the customer's shoe size from an hour ago because that part of the chat was cut to make room.

Think of it like…Copy Anchor Link

Real-world exampleCopy Anchor Link

An online shoe store uses an agent to handle returns. Early in a long chat, the customer says, "I wear size 9, and I want a refund, not store credit." Then they upload photos, ask about shipping, and paste a long order history.

Two hours later, the agent offers store credit and asks for the size again. No code crashed. The early message got lost under a pile of photos and order data. If the chat had grown even longer, the window could fill up, and the program might cut the oldest messages, including that one.

The team fixes it in two ways. First, the agent writes key facts to a small "case notes" record as soon as it learns them. Second, the program puts those notes near the top of every request. The size and refund choice now stay in view, no matter how long the chat gets.

How it worksCopy Anchor Link

Tokens. Models read text as tokens, small chunks of text that can be a word, part of a word, or a single character. Window sizes and costs are counted in tokens, not words.

Who does what. The model is the AI that reads and writes text. The program is your code around it. A call (or request) is one message package your program sends to the model to get one answer. A session is one run or chat, from start to finish. The model plus the program together make the agent.

The window holds everything. The context window is the full set of tokens the model can look at for one answer. Claude's docs say it includes the system prompt (the standing instructions), every message, tool results (what a tool sent back), tool definitions (the list of tools the model may use), and the answer being written. Claude's docs list window sizes from 200,000 to 1 million tokens, depending on the model.

The model is stateless. "Stateless" means it keeps nothing between calls. Claude's docs say you must send the full chat history with every request. So the "memory" is really your program pasting the past back in each time.

That gives three ways an agent forgets:

The third one surprises people. A 2023 study, "Lost in the Middle," found models often did best when the useful fact sat at the start or end of a long input. They did worse when it sat in the middle. Anthropic calls the wider problem context rot: as the window fills, the model gets worse at recalling what is in it.

 Saved notes ──┐
 Instructions ─┤
 Chat history ─┼──► [ context window ] ──► Model ──► Answer
 Tool results ─┘     (limited space)

The diagram shows that the program builds the window fresh on every call. Only what goes in can be "remembered."

How agents keep memory anyway. Anthropic's guide on context engineering describes three main tricks:

1. Compaction — when the window is nearly full, summarize the chat so far and continue from the summary.
2. Note-taking — the agent writes notes to a file or database outside the window, then reads them back later. This gives the agent memory that lasts beyond the window.
3. Sub-agents — helper agents do big jobs in their own clean windows and return only a short summary.

The same guide suggests just-in-time loading: keep short pointers, like file names or links, and fetch the full content only when it is needed.

Common mistakesCopy Anchor Link

• Assuming the model remembers. It keeps nothing between calls, so facts vanish in a new session. Store important facts in notes and load them back.
• Stuffing the window "just in case." Extra text tends to make recall worse, not better. Include only what this step needs, and fetch the rest on demand.
• Burying key facts in the middle. The 2023 study found models missed middle facts more often. Putting goals and key facts near the start or end can help.
• Letting tool results pile up. Big search results and file dumps fill the window fast. Clear or summarize old tool output once it has been used.
• Trusting a summary blindly. Compaction can drop a small but vital detail. Save must-keep facts in notes, not only in the summary.

Remember thisCopy Anchor Link

QuizCopy Anchor Link

Go deeperCopy Anchor Link

• Read first: What is an AI agent?
• Related: The agent loop (think → act → observe)
• Sources are listed below with the date they were last checked.