Analysis: long context did not kill retrieval — it changed what retrieval is for

Every time a context window gets an order of magnitude larger, the same claim circulates: retrieval was a workaround for small windows, the windows are big now, so retrieval is over. Put the documents in the prompt and let the model sort it out.

The claim is not absurd. It is genuinely now possible to place an entire codebase, a year of email, or a stack of contracts into a single request. The reason most production systems still retrieve is not inertia. It is that capacity and utilisation are different things, and that three of retrieval's properties have nothing to do with window size at all.

Capacity is not utilisation

A model that accepts a million tokens will accept a million tokens. Whether it uses them evenly is a separate empirical question, and the answer has been consistent since the phenomenon was first characterised: recall is strongest at the beginning and end of a long input and weakest in the middle.

The practical consequence is uncomfortable. Where a document lands in your prompt affects whether the model uses it, independent of how relevant that document is. A system that concatenates fifty documents in arbitrary order has introduced a quality variable that nobody is monitoring and that will shift the next time the concatenation order changes.

The standard reassurance is a needle-in-a-haystack result — a distinctive sentence planted in a large window and reliably found. That test is much easier than it looks. Finding one unusual string is a lexical matching task. Real questions require aggregating across several passages, noticing that two of them disagree, or determining that the answer is absent. Benchmarks built to test that harder shape consistently show effective working context well below advertised capacity.

The arithmetic nobody wants to do

Cost and latency scale with what you send, not with what you are allowed to send.

Consider a support assistant handling ten thousand questions a day against a 200,000-token knowledge base. Stuffing the base into every request means two billion input tokens a day. At any current frontier price, that is a budget line that gets noticed. Retrieving four thousand relevant tokens per question means forty million — a factor of fifty, applied to every request, forever.

Latency follows the same curve. Time to first token grows with input length, and a support widget where the answer starts appearing in 900 milliseconds is a different product from one that takes eight seconds, regardless of the answer's quality.

Prompt caching complicates this in a way worth stating precisely, because it is the strongest argument on the stuffing side. When a large prefix is identical across many requests, caching can reduce its cost by a large multiple and cut latency significantly. If your corpus is genuinely shared and stable — a fixed product manual, a fixed legal code — this changes the calculation and is under-used. But caching does nothing when the large content differs per user or per request, which describes most enterprise deployments, where the whole point is that each user may see different documents.

The three properties retrieval has that context does not

This is the part of the argument that does not move when windows get bigger.

Attribution. A retrieved passage has an identity before the model ever sees it: a document id, a URL, a page, a revision. When the model cites it, the citation is mechanically checkable. Text dropped into a window has no such identity, and asking a model to cite from an undifferentiated mass produces references that are difficult to verify and easy to fabricate. For any application where a wrong answer needs a traceable source — legal, medical, financial, compliance — this alone settles the design.

Freshness. Corpora change. A retrieval index is updated by writing new rows; a stuffed prompt is updated by rebuilding and resending the whole payload. When your knowledge base changes hourly, one of these is a background job and the other is a per-request tax.

Access control. This is the one that quietly decides most enterprise architectures. Different users are permitted to see different documents. In a retrieval system, permissions are a filter applied at query time, before any text reaches the model — an ordinary, auditable authorisation boundary. In a stuffing system, the permission decision has to happen when assembling the payload, and any mistake means one user's confidential document is inside another user's prompt. Once it is in the window, no downstream control can un-see it.

What long context is genuinely better at

The symmetric error is to conclude that retrieval always wins. Large windows are decisively better for a specific and growing set of tasks.

Whole-artefact reasoning. Summarising a full contract, tracing a bug across a codebase, comparing two long documents clause by clause. Chunking actively destroys the structure these tasks depend on. If the question is about the shape of the whole thing, retrieval gives you fragments of an argument you needed intact.

Agent working memory. An agent forty steps into a task is carrying its own history — tool outputs, intermediate reasoning, files it has read. That is context accumulated within the session, and no retrieval index has it. Larger windows directly extend how long an agent can work before its earlier decisions need compressing.

Low-volume, high-stakes reads. When a task runs a few dozen times a day and getting it right matters more than the cost per call, the simplicity of putting everything in the prompt is worth paying for. Not every system needs an index.

The architecture that actually wins

Stated as a choice, this is a false dichotomy. The strongest current pattern uses both, in sequence.

Retrieve broadly, then read generously. Use retrieval to narrow a large corpus to a candidate set — but where an older system would have passed three 500-token chunks, pass twenty full documents. Retrieval does the job it is good at, which is selection under access control with attribution intact. Long context does the job it is good at, which is reasoning over material without artificial fragmentation.

This inverts an assumption baked into a lot of older RAG code: that chunks should be small because the window is scarce. The window is no longer scarce. Chunk for retrieval quality, not for window budget — retrieve at the granularity that makes matching accurate, then expand each hit to its full section or document before it goes into the prompt.

Two smaller practices matter more than they used to. Order deliberately: given the position effect, the highest-scoring passages should sit at the edges of the assembled context rather than in the middle. And label every passage with its source id in the prompt, so that citation remains mechanical rather than a request for the model's good faith.

Evaluating the right component

One diagnostic habit prevents most wasted effort here. When an answer is wrong, determine whether the necessary document was in the context at all.

If it was not, the retrieval stage failed and no model change will fix it. If it was present and the model still got it wrong, that is a generation problem — and now the question of window size, ordering and prompt structure becomes relevant. Systems that only measure end-to-end quality cannot tell these apart, and teams running them tend to spend months swapping models to fix a broken index.

The short version: long context removed the constraint that made retrieval mandatory, and left every other reason to retrieve exactly where it was. Most production systems will keep both, and the interesting design work has moved from whether to retrieve to how much to hand the model once you have.