Why Your AI Gets Dumber After 100,000 Tokens

Imagine you are sitting a final exam, but instead of the usual rules, your professor says: "Bring everything. Every textbook, every notebook, every loose page of notes you have ever touched." You roll in a trolley stacked four feet high. Surely, you think, this is an overwhelming advantage.

Then the questions begin. The exam hall is dimly lit. None of your books have an index. Several of them contradict each other on the key topics. And you have thirty seconds per question.

Your library is perfectly intact. Your ability to find, connect, and use the right sentence on the right page — in thirty seconds, under pressure — is something else entirely.

This is the exact trap we fall into with AI context windows. A model that accepts a million tokens does not have a million tokens of equally accessible working memory. It has a very large room full of books, and no librarian.

The Cliff Everyone Falls Off

People who work with AI all day keep meeting the same pattern. Up to roughly 50,000 tokens, performance is excellent. Somewhere between 90,000 and 120,000, things go strange: the model misses facts it was clearly given, violates constraints it acknowledged earlier, and produces answers that feel like they came from a smaller, less careful model.

Is the cliff at exactly 100,000 for every model? No. Benchmarks find model-specific curves — some models stumble at 32,000, others hold up past 128,000. What is universal is the shape: performance erodes as context grows, and it erodes faster when the task is harder.

So the useful question is not where the cliff is. It is why there is a cliff at all. There are four reasons, and they compound each other the way bad luck tends to.

Reason 1 — Nobody Reads the Middle of the Book

Read a 500-page novel, come back a week later, and check what you actually remember. Almost certainly the opening chapters and the ending. The long middle stretch has gone blurry.

Language models behave remarkably similarly, and researchers gave the phenomenon a formal name: the Lost in the Middle effect. Place identical evidence at the beginning, middle, or end of a long context, and the model uses it reliably at the extremes — and unreliably in between. Park the critical clue in the middle of a 128,000-token pile, and the model may simply fail to find it, even though it technically processed every single token.

If the most important rule of your project sits on line 3,400 of a giant prompt, do not be surprised when it gets ignored. The model is not being careless. It has the same primacy and recency biases as a human reader — baked in at an architectural level.

Reason 2 — Attention Is a Budget, and the Crowd Keeps Growing

This one needs a two-minute look inside the machine, and it will repay you for years.

As a model processes text, every token looks at every other token and asks: "how relevant are you to my meaning?" The word bank examines its neighbours to decide between a riverbank and a financial institution. This mechanism — self-attention — is the core engine of every modern language model.

The key detail: the relevance scores pass through a function called softmax, which forces them to sum to exactly 1.0. Attention is a fixed budget. You can redistribute it. You cannot expand it.

At 10,000 tokens, that budget of 1.0 is shared among 10,000 candidates. At 100,000 tokens — same budget, ten times the crowd. Picture a talent scout at an open audition with a hard contractual limit of 100 seconds of total attention. Ten singers: ten seconds each. A thousand singers: a tenth of a second each. The genuinely talented singer in row 47 did not get quieter. The scout's ability to pick her out has collapsed.

To be precise: the model does not spread attention evenly — softmax can lock nearly the whole budget onto one obviously relevant token. The real problem is discrimination. A bigger context contains more near-matches, near-contradictions, and plausible impostors. Finding "the 1987 treaty was signed in Quito" is trivial when it is the only treaty in the room. It is genuinely hard when 800 other passages mention other treaties in other capitals.

Reason 3 — The Shortcuts Hiding Inside Long Windows

The third reason is arithmetic, and the arithmetic is dramatic. In standard self-attention, every token compares itself with every other token. A context of n tokens means n × n comparisons — quadratic scaling.

Growth like that turns "fast" into "impossible," so engineers ship shortcuts. Some are honest wins: FlashAttention computes the exact same result with far less memory traffic. Others trade away sight. Sliding windows compare only nearby tokens — so the model genuinely cannot see what fell outside the window. Compression squeezes older context into summaries, and summaries lose detail by definition. Sparse patterns skip comparisons on purpose.

There is no free lunch at a million tokens. Every mechanism that makes a huge window affordable does so by deciding, somewhere, to look at less.

Reason 4 — Trained for Sprints, Entered in a Marathon

The window printed on the spec sheet is not the distance the model was trained at. Training data is overwhelmingly short documents: articles, forum threads, code files, chat logs. Even in a dataset of a trillion tokens, documents genuinely exceeding 100,000 tokens are a tiny minority.

So the circuits for "connect something on line 3 to something on line 3,000" got far less practice than the circuits for local, short-range reasoning. The knowledge is there; the long-distance wiring is thin. Engineers stretch windows further with positional-encoding tricks — rescaling RoPE, techniques like YaRN — and these mostly stop things from breaking, which is not the same as making the model good at two hundred pages.

Giving a model a bigger context window without long-range training is like giving a sprinter a longer track and expecting a marathon. The legs work. The endurance was never trained.

Finding a Fact Is Not the Same as Thinking With It

One more distinction — the one that spec sheets and demos love to blur.

There is a class of tests called needle-in-a-haystack: hide one unusual sentence — "the secret word is persimmon" — inside a very long document, then ask the model to recall it. Models pass this at impressive lengths, and marketing enjoys it. It is also the easiest possible task: one distinctive needle, zero competition.

Real work looks like this: "Here are 200 customer conversations from the past year. Which product categories have recurring complaints, are those complaints getting worse, and which promises did our team make but never keep?" That is hundreds of ordinary, unlabeled needles that must be found, counted, compared, and cross-referenced. Benchmarks that test this kind of multi-step work over long context — BABILong, NoLiMa — show sharp declines as the number of reasoning steps grows.

"Effective context" is therefore not a number hidden in a spec sheet. It is a curve that shifts with the task, with where the evidence sits, with how much noise surrounds it, and with how many steps stand between the question and the answer.

When Your Agent Forgets Its Instructions

People who run agents on long workflows meet a specific, maddening symptom: the model stops following instructions it acknowledged at the start. Formatting rules evaporate. Constraints agreed in hour one get violated in hour three. The persona quietly dissolves. Practitioners call it prompt fade.

The mechanism is ordinary dilution. An instruction at position zero must compete with tens of thousands of newer tokens for influence over the next word. Recency tends to win. If recent conversation implies something different from the original rule, the model follows the implication — not out of disobedience, but arithmetic.

The fix is inelegant and works: re-brief the model near the moment of action. Restate the goal, the hard constraints, and the output format right before you ask for the result — the way a good director gives notes before every take. Not because the actors forgot the play, but because the volume of intervening rehearsal is genuinely large.

You Will Meet This in Your First Week

Everything above happens live inside your sessions with Claude Code, Codex, or Hermes. A session that has been running all afternoon is a 100,000-token context wearing a friendly chat interface. You will recognize the symptoms:

None of this means the tool is broken. It means the session's context has outgrown the model's ability to use it. You cannot fix the architecture. You can absolutely work with it — and the habits are simple.

Six Habits That Beat Attention Decay

• 1
  One task, one session. A fresh session costs nothing; a decayed one costs the afternoon. When a task is finished — or the session starts feeling foggy — close it and start clean.
• 2
  Put decisions in files, not in chat. Project memory files (CLAUDE.md, AGENTS.md, a notes.md) are re-read fresh at the start of every session. Chat history only decays. If a decision matters tomorrow, it belongs in a file today.
• 3
  Hand off between sessions. End long sessions with: "Write a handoff summary: what's done, what we decided, what's next, what to avoid." Start the next session by having the agent read it. Ten lines of summary beat a hundred thousand tokens of history.
• 4
  Re-brief before the big ask. Right before an important request, restate the goal, the constraints, and the required format — even if you already said them this morning. Director's notes, every take.
• 5
  Curate, don't dump. Give the agent the three relevant files, not the whole folder; the failing error message, not the entire log. A focused 8,000-token context beats a bloated 100,000-token one almost every time.
• 6
  Split big jobs into stages. Spec → build → verify, each in its own session with a small, clean context. A chain with checkpoints is easy to debug; one heroic 150,000-token run is not. (If you ever build AI products, these same habits become retrieval, compression, and pipelines — same physics, bigger words.)

Check Yourself

A vast context window is genuinely valuable. It is simply not the same thing as a vast mind. The honest unit of measurement is not tokens accepted — it is relevant evidence reliably used. Until those two quantities converge, the wise engineer treats long context as an archive, and builds the model a small, well-organized desk to think on.

The library is large. The librarian still works best with a short reading list.