gemma4-research/CORPUS_agentic_retrieval.md

Agentic Information Retrieval Beyond RAG, May 2026

A view of where retrieval and memory architectures are converging for agentic applications, with explicit production-vs-experimental sorting for self-hosted small-model deployments (Gemma 4 26B–31B, Qwen3 8–14B).

Last updated: 2026-05-25. Cuts off long-context-as-RAG-replacement, graph retrieval, agent memory, and deep-research-agent patterns.

Headlines

1. The 2023-era named patterns (Self-RAG, CRAG, FLARE, MemGPT) survived as design templates but their original codebases are mostly stale. The 2026 production stack reimplements them as graph nodes inside LangGraph or LlamaIndex Workflows. Use the patterns; don't try to use the original repos.
2. Long context did not replace RAG. Multi-needle retrieval at 1M tokens regressed in some frontier models in 2026; even where it works, the cost/latency math means RAG-then-stuff stays the winning shape. The "just dump the corpus" approach is now a niche choice for single-document deep reasoning, not corpus retrieval.
3. GraphRAG got cost-viable. Microsoft's LazyGraphRAG matches or beats full GraphRAG quality at indexing cost equal to vector RAG and query cost 0.1% of full GraphRAG. GA targeting Q1–Q2 2026 in the MSR repo. For SMB corpora today, LightRAG is the production option.
4. Memory architectures split into two camps: managed CRUD layer (mem0, Zep) vs. agent-edited state (Letta with Context Repositories "MemFS", A-MEM). For small open models, lean toward managed — Gemma 4 will sometimes drop tool calls and you'll lose memory silently if the model owns it.
5. The settled retrieval base layer is hybrid BM25 + dense + RRF fusion + cross-encoder rerank. "Dense embeddings alone" has been retired in serious systems. The lexical signal is load-bearing for error codes, identifiers, and technical corpora that embeddings underperform on.

---

1. Agentic RAG — patterns, not libraries

The original papers all date from 2023. In 2026 they exist as patterns absorbed into agent frameworks.

Pattern	Origin	What survived	What to use
Self-RAG	Asai et al., ICLR 2024	Reflection-token idea (model decides per-step whether to retrieve, critique chunks)	Re-implement as graph nodes; the trained 7B/13B models aren't being updated
CRAG (Corrective RAG)	Yan et al., 2024	retrieve → evaluate → correct → generate	LangGraph cookbook implementation is now the canonical artifact
FLARE	Jiang et al., 2023	Uncertainty-triggered retrieval (low-confidence span ⇒ re-retrieve)	Pattern survives, original repo stale
PRISM / RELOOP	arxiv 2510.14278, 2510.20505 (late 2025)	New, multi-hop with precision-focused control flow	Experimental — track if multi-hop precision is your bottleneck

Frameworks ranked by fit for branching agentic RAG flows:

• LangGraph (graph-native, best for CRAG/Self-RAG-shaped branching)
• LlamaIndex Workflows (event-driven, similar story)
• Haystack (more structured pipelines, weaker on dynamic branching)

GitHub star order, January 2026: LangChain 125K · Dify 114K · RAGFlow 70K · LlamaIndex 46.5K. (source)

Convergence: yes, on the pattern — agent decides when to retrieve, evaluates retrieved chunks, optionally re-queries. No convergence on which framework owns it.

---

2. GraphRAG: the cost story

Microsoft's original GraphRAG had a ~$33K indexing bill on large corpora, making it a non-starter outside well-funded enterprises. 2026 changed that.

Project	Maturity	License	When to use
Microsoft GraphRAG	Production but expensive	MIT	Global-question answering over stable corpora when budget is unconstrained
LazyGraphRAG	Beta, GA Q1–Q2 2026	MIT	The new default — same quality, 0.1% query cost
LightRAG (EMNLP 2025)	Production	MIT	SMB corpora today, Docker-deployable, PG+pgvector+AGE one-DB path
nano-graphrag, fast-graphrag	Experimental	MIT	Hackable baselines, sub-1M-doc cases

2026 consensus on graph vs. vector (per GraphRAG-Bench, ICLR'26): graph wins when (a) coherent domain corpus, (b) questions require synthesizing across many documents, (c) indexing pass is affordable. Vector wins on point-fact lookup, noisy/heterogeneous corpora, cost, latency.

Recommendation: for a self-hosted small-model setup with a coherent domain corpus, LightRAG is the production try-this. Skip full Microsoft GraphRAG (cost). Watch for LazyGraphRAG GA and swap when shipped.

---

3. Long context did not replace RAG

The 2026 reality killed the "just stuff the corpus" dream.

• Claude Opus 4.7 regressed on multi-needle 1M-context retrieval — 32.2% on MRCR v2 8-needle at 1M vs. 78.3% for Opus 4.6. (source)
• Multi-needle leaderboard at 1M tokens: Gemini 3 leads at 89%, GPT-5.5 at 74%, Opus 4.7 at 56%. (source)
• RAGAS 2.0 (April 2026) study found faithfulness dropped up to 40% when context windows were overloaded with irrelevant filler. (source)

Effective context for multi-needle workloads is 200–400K for current frontier models even when the marketed window is 1M+. Above that, attention dilutes, "lost in the middle" kicks in, latency explodes, cost is linear.

Gemma 4 specifically: 128K context. Plenty for the retrieve-then-stuff pattern, but cap effective working context at ~64K to stay above the lost-in-the-middle floor for 8–31B models. Not a corpus replacement.

The winning shape: RAG retrieves the top 50–200K most relevant tokens → long-context model reasons carefully. "Just dump everything" is the niche choice for single long-document analysis (legal contract, research paper).

---

4. Memory architectures

Real movement in 2025–2026. Five-way taxonomy (SurePrompts): provider-managed, self-managing (Letta), CRUD memory layer (mem0), vector RAG, custom in-app schema.

Project	Camp	Maturity	License	Best for
Letta (formerly MemGPT)	Agent-edited state. Feb 2026: Context Repositories ("MemFS") — memory projected into git-backed files operated on via bash/computer-use tools	Production	Apache 2.0	Long-horizon coherence as the product, strong models only
mem0	CRUD memory layer, vector + optional graph, three-level hierarchy	Production	Apache 2.0	"Remember the user across sessions" consumer-app cases
Zep	Graph-based temporal memory, claims 75.14% on LoCoMo (contested vs mem0)	Production	Commercial + community OSS	Temporal recall + graph relationships
A-MEM (arxiv 2502.12110)	Zettelkasten-inspired — adding new memories triggers updates to existing ones, evolving graph	Experimental	Research	If you want to play with self-organizing memory graphs
Hindsight, Memvid, Supermemory, MemPalace	Various	Experimental	Various	Watch, don't commit

For Gemma-4-sized models specifically: lean toward mem0-style managed memory. Letta's self-edit pattern needs a model that reliably calls the memory-save tool every time it should — Gemma 4 won't. The benchmarking fight between mem0 and Zep on LoCoMo is contested; pick on integration fit, not on the leaderboard.

---

5. Retrieval-augmented planning and multi-hop

The 2023-era patterns (ReAct, IRCoT, Self-Ask, DRAGIN) remain the conceptual foundation. Known weaknesses: error propagation, noisy evidence sets, fixed step budgets.

Practical pattern that converged in 2026:

ReAct loop + cross-encoder reranker on every retrieval
         + explicit "do I have enough?" verification step before answering

The verification step is the real change from 2023. Agents are expected to check their evidence rather than just synthesize.

Recent academic:

• PRISM (arxiv 2510.14278) and RELOOP (arxiv 2510.20505) — late 2025 frameworks attempting to fix the precision-recall imbalance of IRCoT-style approaches. Experimental.
• Four-axis design framework survey (arxiv 2601.00536) — useful 2026 organization of the design space.

---

6. Deep-research agents

Heavy activity. OpenAI Deep Research (early 2025) set the template; open-source caught up fast.

Project	Maturity	License	Notes
OpenAI / Google Deep Research	Closed	—	Reference target. OpenAI ~67% GAIA validation
smolagents/open_deep_research	Production code, experimental quality with small models	Apache 2.0	55% GAIA validation with GPT-4o. CodeAgent uses ~30% fewer steps than ToolCallingAgent
gpt-researcher	Production	Apache 2.0	Most mature open option. Planner + search + reader + report writer with citations
LangChain open_deep_research	Production	MIT	Tavily search default, MCP support, model-agnostic
Perplexity Sonar API	Production, closed	—	37% citation hallucination rate (vs ChatGPT Search 67%, Grok 3 94%) per Columbia Journalism Review audit. Best closed-source citation grounding

For Gemma 4 specifically: 7B–13B class models work in deep-research frameworks but quality drops noticeably vs. GPT-4o-class. The 31B variant is probably the sweet spot for self-hosted deep research; smaller variants struggle with the planning step.

---

7. Hybrid / late-interaction retrievers — alive and well

"Dense + reranker won" is not the 2026 consensus.

Approach	Maturity	License	When
ColBERT v2 late-interaction, per-token embeddings	Production	Apache 2.0	Retrieval-heavy reference architecture
ColPali — late-interaction over document images via VLMs, OCR-free	Beta → production transition	Apache 2.0	Visually rich documents (tables, charts, forms). Storage cost is the gotcha (multi-vector per page). Zilliz/Milvus and Weaviate ship integrations
SPLADE — learned sparse, lexical + semantic	Production	Apache 2.0	Hybrid pipelines. Lexical signal load-bearing for technical corpora
BGE-M3 — dense + sparse + multi-vector in one model	Production	MIT	Multilingual default
Qwen3-Embedding (0.6B/4B/8B) — 8B topped MTEB multilingual at 70.58 mid-2025	Production	Apache 2.0	Strong open default, especially multilingual

Settled production pattern (source):

BM25 + dense ANN (parallel) → RRF fusion → top-100 candidates
                                        → cross-encoder rerank → top-5 → LLM

BM25 stubbornly wins on literal matches (error codes, product SKUs, identifiers) — embeddings underperform on those. "Embeddings alone" has been mostly retired in serious systems.

---

8. Reranking and verification

Cross-encoder reranking is table stakes. Choice of model:

Reranker	License	Notes
BGE-reranker-v2-m3	MIT	Multilingual, self-hostable, price/performance default
Cohere Rerank v3.5	Closed/paid	4096-token context, JSON / semi-structured. "Just works" choice
Jina Reranker v3	Apache 2.0	0.6B listwise (query + all candidates in one window). 81.33% Hit@1 at 188ms
Qwen3 Reranker (0.6B/4B/8B)	Apache 2.0	Competitive with BGE/Jina
Voyage rerank-2.5, mixedbread mxbai, nemotron reranker	Various	Also in the mix. Nemotron at 83% Hit@1 / 243ms currently the accuracy leader per AIMultiple benchmark

Verification patterns (genuinely new in 2026):

• Explicit "is this passage relevant?" LLM call after rerank
• Citation-supported answers — every claim must point to a retrieved passage
• Groundedness scores via Ragas

Ragas production thresholds: faithfulness ≥ 0.9, answer relevancy ≥ 0.85, context precision ≥ 0.8.

---

9. Production-vs-experimental quick reference

Thing	Maturity	License	When to use
Hybrid BM25 + dense + RRF + cross-encoder rerank	Production, settled	varies	Default starting point for any new RAG
BGE-reranker-v2-m3 / Jina v3 / Qwen3-Reranker	Production	MIT/Apache	The reranker step above
LangGraph / LlamaIndex Workflows	Production	MIT	Implementing CRAG / Self-RAG–shaped flows
gpt-researcher / smolagents open_deep_research	Production (open)	Apache/MIT	Deep-research agents
LightRAG	Production for SMB corpora	MIT	Graph-aware retrieval without MSR cost
LazyGraphRAG	Beta (Q1–Q2 2026 GA)	MIT	Replacing full GraphRAG when shipped
mem0	Production	Apache 2.0	"Remember the user" memory
Letta + Context Repositories (MemFS)	Beta–Production	Apache 2.0	Long-horizon agent state, strong models only
ColPali	Beta–production	Apache 2.0	Visually rich documents, accept storage cost
Long context as RAG-replacement	Niche	n/a	Single long document, not corpus
Self-RAG original models, FLARE original repo	Paper-only	research	Read the paper, re-implement the pattern
A-MEM	Experimental	research	Self-organizing memory graph experiments
PRISM / RELOOP	Experimental	research	Multi-hop precision experiments

Genuine convergence:

• Hybrid retrieval + cross-encoder rerank is the universal base.
• CRAG-shaped self-correction is the dominant agentic RAG pattern.
• Long context complements RAG; doesn't replace it.
• Citations / grounded answers are expected.

Still no consensus on:

• Which agent framework wins (LangGraph vs LlamaIndex Workflows vs smolagents vs roll-your-own).
• Memory: managed CRUD layer vs agent-edited state.
• When graph beats vector for a given corpus (domain-dependent).
• Whether tool-call agents or code-writing agents are the better substrate (smolagents data favors code; production deployments split).
• Memory benchmarking — LoCoMo numbers actively contested.

---

10. Picking a stack for a Gemma 4 agent

For a self-hosted Gemma 4 26B/31B agent that already does basic tool-calling + embedding RAG and wants to step up:

1. Base retrieval: hybrid BM25 + dense (Qwen3-Embedding-4B or BGE-M3) with RRF fusion, then BGE-reranker-v2-m3 cross-encoder. Non-experimental but the foundation everything else assumes.
2. Agentic layer: CRAG-shaped flow in LangGraph — evaluate retrieved chunks, fall back to web search if score is low, verify before answering. Skip Self-RAG-the-models (won't work with Gemma); use the pattern.
3. Memory: mem0 for user-facing recall. Skip Letta unless the agent specifically needs long-horizon autonomous behavior — Gemma 4 will sometimes drop tool calls and you'll lose memory silently if the model owns it.
4. Web search: gpt-researcher or smolagents/open_deep_research, with Gemma 4 31B doing the planning/synthesis. Expect 30–40% lower quality than GPT-4o-class; budget for it.
5. Graph experiment: LightRAG on whichever surface has a coherent domain corpus. Skip full MSR GraphRAG (cost). Watch for LazyGraphRAG GA and swap.
6. Long context: don't lean on it. 128K Gemma 4 context is for "retrieve a lot then reason," not "skip retrieval." Cap effective working context at ~64K.
7. Worth a one-off experiment: ColPali on any visual-document corpus (genuinely different OCR-free path), A-MEM for memory if mem0 feels too static, PRISM-style multi-hop if you have a benchmark question your CRAG flow can't handle.

Don't spend time on: original 2023 implementations (Self-RAG models, FLARE repo, MemGPT pre-Letta). The ideas survived; the codebases did not.