The Retrieval Shift: Why RAG Is Not Dead, But It Is Evolving

Consider a common architecture-review scenario (an illustrative composite, drawn from patterns I see repeatedly in retrieval design reviews): a team is building a conversational AI voice agent. The kind where response latency matters in hundreds of milliseconds, because a user on the other end of a phone call notices every pause. The architecture they start with is textbook: embed the knowledge base, store it in a vector database, retrieve the top-k chunks at query time, then pass them to the model for generation.

It works. But it is slow. A retrieval round-trip can add a few hundred milliseconds per turn. For a voice agent, that gap between question and answer feels like an eternity.

So they try something different. The knowledge base is small enough (on the order of 80K tokens of product documentation) that the entire thing fits into the context window. No retrieval step, no vector database, no embedding pipeline. Just the model and its context.

The result is often counterintuitive. Not only can it be faster, the answers can be better. The model reasons across the full document set instead of working with five decontextualized chunks. For that specific kind of use case, the “just stuff it in context” approach can win on every metric that matters.

That pattern is exactly the kind of anecdote fueling a narrative that has echoed through every AI conference and Slack channel for the past year: RAG is dead.

It is also exactly the kind of anecdote that leads teams astray when they try to generalize it.

The “RAG Is Dead” Narrative

The argument is straightforward. Context windows have grown dramatically over the past three years:

• May 2023: Anthropic announces a 100K token context window for Claude (May 11, 2023), among the first to break the 32K barrier at scale. The original Claude had launched two months earlier with a much smaller window.
• February 2024: Google announces Gemini 1.5 Pro with 1 million tokens. The AI community collectively asks: why build retrieval when you can dump everything into context?
• June 2024: Google opens Gemini 1.5 Pro’s 2M token window to all developers through the API.
• April 2025: OpenAI’s GPT-4.1 arrives with a 1M token context window. Anthropic’s Claude models follow with 1M tokens at flat-rate pricing, no surcharge for using the full window.

The logic seems airtight. If your entire knowledge base fits in the context window, why maintain the complexity of an embedding pipeline, a vector database, a retrieval layer, and a reranking step? Simpler architectures are better architectures. Ship the context, skip the plumbing.

But this framing conflates two different things: what the model accepts and what the model reasons well over. That distinction matters enormously in production.

What the Research Actually Shows

The foundational work here is Liu et al. (2023), “Lost in the Middle.” The Stanford and Berkeley researchers found a U-shaped performance curve: language models perform best when relevant information sits at the very beginning or end of the input, and performance degrades significantly when the answer is buried in the middle. This held true across multi-document question answering and key-value retrieval tasks, and it held true even for models explicitly designed for long contexts.

Three years later, the problem has not gone away. It has gotten more nuanced.

Chroma’s Context Rot research tested 18 LLMs across Anthropic, OpenAI, Google, and Alibaba model families. The results showed that performance grows “increasingly unreliable as input length grows.” One counterintuitive finding: models actually performed worse when the input preserved a logical flow of ideas. Shuffled haystacks outperformed structured ones across all 18 models, suggesting that coherent context can mislead models into false confidence about their comprehension.

Perhaps the most uncomfortable finding comes from Du et al. (2025). Their research demonstrated that LLM performance degrades 13.9% to 85% as input length increases, even when models can perfectly retrieve all relevant information. This is not a retrieval failure. It is a reasoning failure. The context length itself is the limiting factor, independent of whether the model can “see” the relevant tokens.

For practitioners: The effective context length of most models sits at roughly 50-65% of the marketed capacity. A model claiming 200K tokens typically becomes unreliable around 130K. Some users have reported a sharper falloff: one community thread on the Gemini CLI describes contextual memory degrading after roughly 20% of the context window is in use. That is an anecdotal, self-reported observation rather than a vendor benchmark, but it points the same direction: the number on the spec sheet is not the number you can rely on.

I wrote about context quality as a first-class architectural concern last week. The research reinforces the same principle: more context is not better context. Curated, relevant context outperforms exhaustive context, sometimes by a wide margin.

The Economics Nobody Discusses

Set aside performance for a moment. Let us talk about money.

Here is a worked example using current Claude Sonnet 4.6 pricing ($3 per million input tokens):

Approach	Tokens Sent per Query	Cost per Query	10,000 Daily Queries
RAG (top-5 chunks)	~4,000	$0.012	$120/day
Full context (200K)	200,000	$0.60	$6,000/day
Full context + prompt caching	200,000 (cached)	~$0.06	$600/day

Source: Cost model derived from Anthropic pricing and MindStudio’s analysis.

The raw numbers are stark. Full-context loading costs 50x more per query than targeted retrieval. Prompt caching (where cache reads cost 10% of the standard input price) narrows this dramatically, bringing it down to 5x. That is a meaningful improvement, but for a production system handling tens of thousands of queries daily, the difference between $120 and $600 per day compounds fast.

Now consider the other side of the ledger. RAG is not free either. A production RAG system carries hidden infrastructure costs: vector database hosting ($50-300/month), embedding API costs for initial indexing and ongoing updates, re-embedding when documents change, and engineering time for retrieval tuning. For an internal tool with a handful of users querying the same stable documents hundreds of times daily, the total cost of owning a RAG pipeline can exceed the token cost of a cached long-context approach.

The economics are not one-sided. They depend on three variables: corpus size, query volume, and update frequency.

What this looks like in practice. If your knowledge base is under 100K tokens, relatively static, and queried by a small team, long context with prompt caching is likely cheaper and simpler. If your corpus exceeds 500K tokens, changes frequently, or serves thousands of concurrent users, RAG remains the more cost-effective architecture by a significant margin. The crossover point shifts with each pricing update, so the right answer is to model your specific workload, not to follow a universal rule.

Google charges differently, which changes the calculus further. Gemini 2.5 Pro pricing doubles from $1.25 to $2.50 per million input tokens once you exceed 200K tokens. There is no flat-rate long-context window here. This price structure actively incentivizes retrieval for larger contexts.

The Architecture Shift

Here is what the “RAG is dead” crowd gets right, even if their conclusion is wrong: the retrieval layer is evolving. The change is not that retrieval disappears. It is that retrieval moves closer to the model.

Phase 1: External Vector Databases (2022-2024). Pinecone, Weaviate, Qdrant, Milvus. Dedicated infrastructure, separate from the application. The model sends a query to an external service, waits for results, then generates. This architecture works, but it adds latency, operational complexity, and another service to monitor.

Phase 2: Embedded and In-Memory Vector Search (2024-2026). A new generation of tools eliminates the network hop:

• LanceDB runs embedded inside your application process. Built on the Lance columnar format (a Rust core that is 100x faster than Parquet for random access), it works directly on S3 without a persistent server. AWS published an architecture for scaling to 1B+ vectors using LanceDB with Lambda, no dedicated database instance required.
• Turbopuffer takes an object-storage-first approach. Cold storage at ~$0.02/GB on S3, with NVMe SSD caching for warm queries. It uses clustered indexes instead of HNSW graphs, trading some recall for dramatically lower storage costs.
• SQLite-vec is a pure C extension by Alex Garcia that adds vector search to SQLite. It runs everywhere: laptops, mobile devices, browsers via WASM, Raspberry Pis. Sub-75ms query times for dimensions up to 1024.

The broader trend is that vectors are moving from a “database category” to a data type within existing databases. PostgreSQL added pgvector and pgvectorscale. MongoDB integrated Atlas Vector Search. Oracle shipped vector support in Database 23ai. You no longer need a separate vector database; you need vector capabilities in the database you already run.

Phase 3: Retrieval as Native Model Operation (emerging). This is where things get speculative but directional. As context windows grow and models internalize more knowledge, the boundary between “retrieval” and “reasoning” blurs. Prompt caching is an early signal: the model provider stores your context server-side and reads from it at 10% of the input cost. This is, functionally, a retrieval cache managed by the model provider rather than by your infrastructure. The trajectory points toward models that handle their own retrieval internally, with the developer providing a knowledge source rather than a pre-retrieved context.

A Decision Framework

After reviewing the research, the economics, and the architecture trends, here is when to use each approach. This framework is my own synthesis of the studies cited above, the cost model below, and the architecture shift, cross-checked against practitioner decision guides.

Factor	Use Long Context	Use RAG	Use Hybrid
Corpus size	Under 100K tokens	Over 500K tokens	100K-500K tokens
Update frequency	Static or rarely changed	Daily or real-time updates	Weekly updates
Query type	Bounded reasoning (contract review, document comparison)	Search and synthesis across large corpora	Facts from retrieval, reasoning over context
Query volume	Low (internal tools, <100/day)	High (production, 1,000+/day)	Medium to high
Latency tolerance	Can accept 1-3s response times	Needs sub-second responses	Depends on use case
Budget	Token costs acceptable; want to avoid infra overhead	Need cost control at scale	Willing to invest in both

A real-world example of the “long context without RAG” approach: Andrej Karpathy recently described building personal knowledge bases as LLM-compiled wikis. At roughly 100 articles and 400K words, he found that the LLM handles retrieval with auto-maintained index files and summaries, no vector search needed. His system collects raw sources, compiles them into structured markdown, and then queries against the full knowledge base. “I thought I had to reach for fancy RAG,” he wrote, “but the LLM has been pretty good about auto-maintaining index files.” This works at his scale. It would not work at 10 million tokens or with real-time updates, which is precisely where the decision framework above applies.

The hybrid pattern is worth emphasizing. The intuition is that using vector retrieval to identify relevant context, then loading those results into a long context window for reasoning, captures the strengths of both: precise selection plus broad reasoning. One vendor blog reports a single European-bank case study where the hybrid approach beat either method alone across most enterprise use cases, with RAG stronger on cross-document synthesis and long context stronger on simple, bounded queries. Treat those figures as directional and self-reported, not as an independently verified benchmark: no primary dataset or peer-reviewed study sits behind them. The underlying logic, though, is consistent with the research above and with how the two methods fail differently.

How to build the check. Before committing to an architecture, profile your actual workload. Measure corpus size in tokens, not pages. Calculate cost per query at your expected volume using current API pricing. Test retrieval quality: does your RAG pipeline surface the right chunks? Test context quality: does the model reason accurately over your full document set, or does performance degrade past a certain length? The answer is almost always “it depends,” and the variables are measurable.

This mirrors what I found with the Economy criterion in context engineering: the cheapest token is the one you never send. Retrieval, at its best, is a mechanism for selecting the tokens that matter and excluding the ones that do not. That function does not become less valuable as context windows grow. It becomes more valuable, because the cost of sending everything grows with it.

Where This Is Heading

The voice agent scenario above is a real pattern. For a small, stable knowledge base with latency-sensitive requirements, loading everything into context can be the right call. But that use case represents a narrow slice of production AI workloads.

The broader trajectory is not “RAG dies.” It is that retrieval architectures are converging with the model layer. External vector databases are giving way to embedded search. Embedded search will give way to model-native retrieval operations. The retrieval problem does not disappear when context windows grow; it moves.

If anything, larger context windows make the quality of what enters the context more important, not less. A million tokens of noisy, partially relevant content will underperform ten thousand tokens of precisely retrieved, validated information. That is what the research consistently shows, and it is what production experience confirms.

The teams that will build the best systems in the next two years are not the ones picking a side in the “RAG vs. long context” debate. They are the ones treating retrieval as a design dimension with multiple valid configurations, choosing the right point on the spectrum for each workload, and evolving their architecture as both models and economics shift underneath them.

RAG is not dead. Retrieval is just getting closer to the model. And the organizations that understand this shift will spend less, ship faster, and build systems that actually work at scale.

Priority	Action	Why It Matters
This week	Profile your largest RAG workload: measure corpus size in tokens, query volume, and cost per query	You cannot optimize what you have not measured. Many teams are over-engineering or under-engineering their retrieval layer.
This month	Test prompt caching on your highest-volume, lowest-change knowledge base	Caching at 90% discount may eliminate the need for a vector database on stable corpora.
This quarter	Evaluate an embedded vector search tool (LanceDB, pgvector, or SQLite-vec) for one workload	Eliminating the network hop to an external vector DB reduces latency and operational complexity.
Ongoing	Build a hybrid retrieval pipeline for your most complex use case: RAG for fact retrieval, long context for reasoning	RAG and long context fail in different ways, so combining them tends to cover each other’s weak spots. Validate the gain on your own workload.