TL;DR
- “Bigger context windows kill RAG” is a myth. At enterprise scale you always have to choose what’s relevant – and choosing what’s relevant is retrieval.
- Pasting everything into the prompt fails three ways: cost/latency balloon, models get “lost in the middle,” and your data never fits anyway.
- Retrieval isn’t disappearing – it’s evolving from a mechanical lookup into agentic retrieval (plan → search → rerank → answer), and ultimately into memory.
- The model provides reasoning; the retrieval layer provides memory. Neither replaces the other.
- The scarce skill is shifting from writing retrieval plumbing to deciding what a system should remember, forget, and prioritize – a judgment call, not a coding one.
Every few months, someone declares RAG dead. The argument sounds convincing: context windows are exploding – millions of tokens now – so why bother retrieving the right documents when you can just paste everything into the prompt and let the model sort it out?
It’s a tidy story. It’s also wrong. But understanding why it’s wrong tells you where enterprise AI is actually heading – and it isn’t where the headlines point.
Let’s start with what RAG is, plainly. Retrieval-augmented generation is the pattern behind most enterprise AI you’ve used: before the model answers, a search step pulls the handful of relevant documents from your data and hands them to the model as context. It’s how a chatbot answers questions about your policies instead of the whole internet. Simple idea. Enormously useful.
The “bigger window” mirage
Here’s the seductive claim: if the model can read a million tokens, just give it everything. No retrieval, no search index, no complexity.
Try it at enterprise scale and three problems show up fast.
Cost and latency scale with what you stuff in. Every token in the window is paid for and processed on every single call. Pasting a 500-page knowledge base into each question isn’t clever – it’s a way to turn a one-cent query into a one-dollar one, and a fast answer into a slow one.
Models get lost in the middle. Give a model a haystack and it reliably finds facts at the start and end of the context – and quietly misses the ones buried in the middle. More context is not more understanding.
And your data doesn’t fit anyway. “A million tokens” sounds infinite until you remember a mid-size company’s documents run to billions. You will always need to choose what’s relevant. Choosing what’s relevant is retrieval. You never escaped it.
So no – a bigger window doesn’t kill RAG. It just changes RAG’s job.
Retrieval grows up: from lookup to memory.
From retrieval to memory
Here’s the shift worth paying attention to. Early RAG was mechanical: match the query to some text chunks, staple them to the prompt, hope for the best. It broke on vague questions, multi-step questions, and anything that needed reasoning about what to look for.
The next generation – Microsoft ships it as agentic retrieval in Azure AI Search – does something more human. It plans. It breaks a complex question into sub-questions, runs several searches, weighs and reranks what comes back, and assembles an answer with citations. Retrieval stops being a lookup and becomes a reasoning loop.
Follow that arc and you can see where it lands: retrieval is becoming memory. Not a filing cabinet the model checks, but something closer to how you recall things – fetching what’s relevant, in context, when you need it, and ignoring the rest. The model provides the reasoning; the retrieval layer provides the memory. Neither replaces the other. That partnership is the actual future – and it’s a more durable idea than any single context-window record.
A real scenario: the insurance policy assistant
Make it concrete. Picture a large insurer whose support team fields questions like: “For claim types like this one, what’s our coverage position when the policy was amended mid-term and the incident spans two periods?” (An illustrative scenario, not a specific customer.) That’s not a keyword lookup. Answering it means pulling the master policy, the mid-term endorsement, the claims-handling guideline, and the relevant regulation – then reasoning across all four.
Naive RAG chokes on this. It matches “coverage” and “claim,” staples a few chunks to the prompt, and produces a confident, incomplete, sometimes wrong answer. Agentic retrieval handles it the way a senior adjuster would: it decomposes the question (“What’s the base coverage? What did the endorsement change? What does the guideline say? Does regulation override?”), runs a search for each, reranks what comes back, and assembles an answer with citations to the exact clauses. Same underlying documents – a fundamentally better result, because retrieval started reasoning about what to look for. That is the difference between “search that finds text” and “memory that answers questions.”
Route by difficulty – pay for reasoning only when reasoning is required.
When to use which
Agentic retrieval isn’t a blanket upgrade – it’s a tool you route to. The rule of thumb:
- Plain retrieval for simple, single-fact lookups (“What’s the deductible on policy X?”) – fast and cheap.
- Agentic retrieval for vague, multi-step, or reasoning-heavy questions – worth the extra model calls.
- A router in front that classifies question difficulty and sends each to the right path – so you pay for reasoning only when reasoning is required.
The honest challenges
If you’re building on this, go in clear-eyed:
- Agentic retrieval costs more per query. Planning and multiple searches mean more model calls. It’s worth it for hard questions – and overkill for simple ones. Route accordingly.
- Evaluation gets harder. When retrieval reasons, “why did it answer that?” has more moving parts. You need traceable citations and a test set of real questions, not vibes.
- Freshness and permissions still bite. Memory is only as good as what it can see. Your index has to stay current and honor who’s allowed to see what – retrieval must respect the same access rules as the source.
- It’s moving fast. Some agentic-retrieval capabilities are generally available as of the 2026-04 API; others remain in preview. Check what’s GA before you depend on it.
What this asks of the people building it
There’s a human shift hiding in this technical one. For a decade, the scarce skill was writing the retrieval plumbing – chunking, embeddings, vector math. As the platform absorbs that, the scarce skill moves up a level: deciding what the system should remember, forget, and prioritize.
That’s not an engineering question anymore. It’s an editorial one – closer to judgment than to code. The teams who win the next phase won’t be the ones with the cleverest chunking strategy. They’ll be the ones who are wisest about what deserves to be remembered, and honest about what the machine should never be trusted to recall on its own.
Where to start
A practical path that won’t age badly:
- Don’t rip out RAG. If you have working retrieval, you’re positioned well – the shift is an upgrade, not a teardown.
- Split your questions by difficulty. Send simple lookups to plain retrieval; reserve agentic retrieval for the genuinely hard, multi-step ones.
- Instrument citations from day one. If you can’t trace why the system answered, you can’t trust or improve it.
- Treat your index as a living memory. Budget for freshness and access control, not just the initial load.
- Decide what to forget. Deliberately exclude the stale and the sensitive. Memory is a choice, not a dump.
FAQ: clearing up the RAG confusion
“Won’t bigger context windows eventually make retrieval unnecessary?” No. Even with a million-token window, enterprise data runs to billions of tokens, so you always select what’s relevant – and selecting relevance is retrieval. Bigger windows change retrieval’s job; they don’t remove it.
“Is agentic retrieval just RAG with extra steps?” It’s RAG that reasons. Instead of one search, it plans sub-questions, runs several searches, and reranks – which is exactly what hard, multi-step questions need. For simple lookups, it is unnecessary overhead, which is why you route by difficulty.
“Do we need to rebuild our RAG system?” Usually not. If you have working retrieval and a clean index, agentic retrieval is an upgrade on top, not a teardown.
“Does ‘retrieval as memory’ mean the AI remembers everything?” That’s the trap. Good memory is selective – it deliberately forgets the stale and excludes the sensitive. What to forget is a design decision, not an accident.
“Is this production-ready or still a preview?” Both, depending on the feature. As of the 2026-04 API, some agentic-retrieval capabilities are generally available and others remain in preview – confirm what’s GA before you build a dependency on it.
The takeaway
“RAG is dead” makes a great headline and a terrible strategy. Retrieval isn’t going away – it’s growing up, from a mechanical lookup into something that reasons, and eventually into memory itself.
The larger pattern is the one to hold onto: as models get more capable, the value doesn’t move into the model – it moves to the judgment around it. What to remember. What to surface. What to leave out. Those are human decisions dressed as technical ones, and they’re getting more important, not less.
Long live RAG – not the version we started with, but the one it’s becoming.
Go deeper
- Retrieval-augmented generation (RAG) in Azure AI Search – Microsoft Learn: the pattern, its real challenges, and the retrieval components that matter.
- Agentic retrieval in Azure AI Search – Microsoft Learn: query planning, multi-step search, and what’s GA vs preview as of the 2026-04 API.
Originally published at next2know.blog. Harry Arce is a Data & AI Advocate, Microsoft Digital Solutions Engineer, and Postgrad Data and AI Instructor. He writes “Data & AI – Applied” – data in, intelligence out.
Leave a Reply