AI Context Window: Why Your Assistant Forgets Things

Your AI assistant nails the first three questions. On the fourth it asks for the customer's name again, the one it already had. Owners around Burnaby and New Westminster ask us about this constantly, usually phrased as "is it broken?"

It isn't broken. It hit the edge of its context window. Once you know what that limit actually covers, most of the strange behaviour stops being mysterious.

A context window is working memory, not a filing cabinet

Anthropic's documentation defines it plainly. The context window "refers to all the text a language model can reference when generating a response, including the response itself," and it "represents a 'working memory' for the model" (Anthropic, Context windows).

Here's the part that surprises people. The model keeps nothing between turns. OpenAI's docs say it directly: "each text generation request is independent and stateless," and to keep a conversation going you "include the model's previous response output as input, and append that input to your next request" (OpenAI, Conversation state).

So the assistant isn't losing a memory. Your software is re-sending the entire conversation every turn, and the context window is the ceiling on how much it can hand over at once.

Everything counts, not just what you typed

Anthropic is specific about what fills the window: "the system prompt, every message in messages (including tool results, images, and documents), and your tool definitions. The output Claude generates for the turn, including its extended thinking, counts too."

In practice the window fills from places nobody sees:

• The system prompt, brand voice rules, and policy text you wrote once and forgot about
• Every tool definition, including tools that never get called
• Raw tool output, like a full CRM record when you needed one phone number
• Attached files. Anthropic's own estimate puts a 500 kB research paper at roughly 125,000 tokens and an average 10 kB web page at about 2,500 (Anthropic, Pricing)
• The model's own replies, piling up turn after turn
• Its internal reasoning tokens, on models where those stay in the history

As a rough conversion, Anthropic puts one token at "approximately 4 characters or 0.75 words in English." A long email thread is a few thousand tokens. A day of call transcripts is far more.

Two very different ways it breaks

Hard overflow. When the input alone is too big, Anthropic's API returns a 400 error reading "prompt is too long." OpenAI warns that "tokens generated in excess of the context window limit may be truncated in API responses." In chat products the failure is quieter. Anthropic notes that chat interfaces such as claude.ai "can also manage the context window on a rolling 'first in, first out' basis." The oldest turns fall off the front. That is your literal mid-conversation amnesia, and nothing warns you it happened.

Soft degradation. This one is sneakier, because nothing fails. Anthropic's docs put it bluntly: "As token count grows, accuracy and recall degrade, a phenomenon known as context rot."

Peer-reviewed work backs the shape of the problem. In "Lost in the Middle" (Liu et al., TACL 2024), performance "is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models" (ACL Anthology).

Google's Vertex AI documentation reports something similar once you ask for more than one fact at a time. Gemini "comes standard with a 1-million-token context window" with "near-perfect retrieval (>99%)," but "in cases where you might have multiple 'needles' or specific pieces of information you are looking for, the model does not perform with the same accuracy" (Google Cloud, Long context).

A bigger window is not the fix

Windows did get much bigger. As of August 2026, Anthropic's documentation lists a 1M-token context window for Claude Opus 5, Opus 4.8, Opus 4.7, Opus 4.6, Sonnet 5, and Sonnet 4.6, while other models including Sonnet 4.5 sit at 200k tokens. Long context carries no surcharge: "A 900k-token request is billed at the same per-token rate as a 9k-token request."

Same rate still means more tokens, every turn. Anthropic's published price for Claude Opus 5 as of August 2026 is $5 per million input tokens and $25 per million output tokens. Run the arithmetic and a conversation carrying 500,000 tokens of history costs roughly $2.50 in input on every turn, before the model says a word. That figure is an illustrative calculation from the published rate, not a quoted price.

Caching is the lever most teams skip. Anthropic prices a cache hit at 0.1x the base input price, with a five minute cache write at 1.25x, so stable context pays for itself after about one read.

What we build instead

Curation beats capacity, and the patterns that survive production are unglamorous. Pull the three relevant records instead of the whole database. Summarize old turns rather than replaying them. Keep durable facts like customer, job number, and quoted price in your CRM, then re-inject only those. Put the instructions that matter at the top and the live question at the bottom. Start a clean thread per job instead of running one endless chat. Vendors ship help for this now, including server side compaction, though Anthropic is careful to note that cached prefixes "still occupy the context window."

Where this doesn't apply

If your assistant handles five-turn conversations, none of this is your bottleneck, and bolting on a retrieval layer is wasted money. Answer the question, book the job, done.

Sources also disagree on how bad long context really is, and honest reading means saying so. Google reports better than 99% single fact retrieval at a million tokens. Liu et al. found sharp position effects across the models they tested in 2024. Both are probably right about their own setups. Treat vendor retrieval numbers as a best case on clean data, then test on your own messy transcripts before trusting them.

There's a privacy limit too. A bigger window tempts you to paste in everything. Canada's federal, provincial and territorial privacy authorities, in principles adopted December 7, 2023, ask organizations to "use anonymized or de-identified information within prompts to a generative AI system rather than personal information" where possible, and to keep retention schedules covering "system prompts, and outputs" (Office of the Privacy Commissioner of Canada). That's a design constraint, not a footnote.

Plenty of businesses are also right to wait. Statistics Canada found that in the second quarter of 2026, 19.2% of Canadian businesses reported using AI to produce goods or deliver services over the preceding 12 months, up from 6.1% in the second quarter of 2024, while 40.0% said AI is not relevant to their business (Statistics Canada). If you're in that 40%, context windows are somebody else's problem for now.

Everything above was checked against current vendor documentation in August 2026. Model names, window sizes and prices move fast, so re-check them before you budget on them.

Sources

• Anthropic. "Context windows." Claude Platform Docs, accessed August 2026. Link
• Anthropic. "Pricing." Claude Platform Docs, accessed August 2026. Link
• OpenAI. "Conversation state." OpenAI API documentation, accessed August 2026. Link
• Google Cloud. "Long context." Vertex AI generative AI documentation, accessed August 2026. Link
• Liu, Nelson F., Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. "Lost in the Middle: How Language Models Use Long Contexts." Transactions of the Association for Computational Linguistics, vol. 12 (2024), pp. 157 to 173. Link
• Statistics Canada. "Analysis on artificial intelligence use by businesses in Canada, second quarter of 2026." Released June 11, 2026. Link
• Office of the Privacy Commissioner of Canada and federal, provincial and territorial privacy authorities. "Principles for responsible, trustworthy and privacy-protective generative AI technologies." Adopted December 7, 2023. Link

Autana Solutions builds AI employees for small and mid-sized businesses across Metro Vancouver, and context design is most of the real work. If your assistant keeps losing the thread halfway through, book a free call and we'll map exactly where the window is filling up.