gemma4-research/CORPUS_agentic_retrieval.md

# Agentic Information Retrieval Beyond RAG, May 2026

> A view of where retrieval and memory architectures are converging for
> agentic applications, with explicit production-vs-experimental sorting
> for self-hosted small-model deployments (Gemma 4 26B–31B, Qwen3 8–14B).
>
> Last updated: 2026-05-25. Cuts off long-context-as-RAG-replacement,
> graph retrieval, agent memory, and deep-research-agent patterns.

## Headlines

1. **The 2023-era named patterns (Self-RAG, CRAG, FLARE, MemGPT) survived
   as design templates but their original codebases are mostly stale.**
   The 2026 production stack reimplements them as graph nodes inside
   LangGraph or LlamaIndex Workflows. Use the patterns; don't try to use
   the original repos.
2. **Long context did not replace RAG.** Multi-needle retrieval at 1M
   tokens regressed in some frontier models in 2026; even where it works,
   the cost/latency math means RAG-then-stuff stays the winning shape.
   The "just dump the corpus" approach is now a niche choice for
   single-document deep reasoning, not corpus retrieval.
3. **GraphRAG got cost-viable.** Microsoft's
   [LazyGraphRAG](https://www.microsoft.com/en-us/research/blog/lazygraphrag-setting-a-new-standard-for-quality-and-cost/)
   matches or beats full GraphRAG quality at indexing cost equal to vector
   RAG and query cost 0.1% of full GraphRAG. GA targeting Q1–Q2 2026 in
   the MSR repo. For SMB corpora today, [LightRAG](https://github.com/hkuds/lightrag)
   is the production option.
4. **Memory architectures split into two camps**: managed CRUD layer
   (mem0, Zep) vs. agent-edited state (Letta with Context Repositories
   "MemFS", A-MEM). For small open models, lean toward managed — Gemma 4
   will sometimes drop tool calls and you'll lose memory silently if the
   model owns it.
5. **The settled retrieval base layer is hybrid BM25 + dense + RRF
   fusion + cross-encoder rerank.** "Dense embeddings alone" has been
   retired in serious systems. The lexical signal is load-bearing for
   error codes, identifiers, and technical corpora that embeddings
   underperform on.

---

## 1. Agentic RAG — patterns, not libraries

The original papers all date from 2023. In 2026 they exist as patterns
absorbed into agent frameworks.

| Pattern | Origin | What survived | What to use |
|---------|--------|---------------|-------------|
| **Self-RAG** | Asai et al., ICLR 2024 | Reflection-token idea (model decides per-step whether to retrieve, critique chunks) | Re-implement as graph nodes; the trained 7B/13B models aren't being updated |
| **CRAG (Corrective RAG)** | Yan et al., 2024 | retrieve → evaluate → correct → generate | LangGraph cookbook implementation is now the canonical artifact |
| **FLARE** | Jiang et al., 2023 | Uncertainty-triggered retrieval (low-confidence span ⇒ re-retrieve) | Pattern survives, original repo stale |
| **PRISM / RELOOP** | arxiv 2510.14278, 2510.20505 (late 2025) | New, multi-hop with precision-focused control flow | Experimental — track if multi-hop precision is your bottleneck |

**Frameworks ranked by fit for branching agentic RAG flows:**

- **LangGraph** (graph-native, best for CRAG/Self-RAG-shaped branching)
- **LlamaIndex Workflows** (event-driven, similar story)
- **Haystack** (more structured pipelines, weaker on dynamic branching)

GitHub star order, January 2026:
LangChain 125K · Dify 114K · RAGFlow 70K · LlamaIndex 46.5K.
([source](https://florinelchis.medium.com/top-10-rag-frameworks-on-github-by-stars-january-2026-e6edff1e0d91))

**Convergence:** yes, on the pattern — agent decides when to retrieve,
evaluates retrieved chunks, optionally re-queries. **No convergence** on
which framework owns it.

---

## 2. GraphRAG: the cost story

Microsoft's original GraphRAG had a ~$33K indexing bill on large corpora,
making it a non-starter outside well-funded enterprises. 2026 changed
that.

| Project | Maturity | License | When to use |
|---------|----------|---------|-------------|
| [Microsoft GraphRAG](https://github.com/microsoft/graphrag) | Production but expensive | MIT | Global-question answering over stable corpora when budget is unconstrained |
| [LazyGraphRAG](https://www.microsoft.com/en-us/research/blog/lazygraphrag-setting-a-new-standard-for-quality-and-cost/) | Beta, GA Q1–Q2 2026 | MIT | The new default — same quality, 0.1% query cost |
| [LightRAG](https://github.com/hkuds/lightrag) (EMNLP 2025) | Production | MIT | SMB corpora today, Docker-deployable, PG+pgvector+AGE one-DB path |
| [nano-graphrag](https://github.com/gusye1234/nano-graphrag), [fast-graphrag](https://github.com/circlemind-ai/fast-graphrag) | Experimental | MIT | Hackable baselines, sub-1M-doc cases |

**2026 consensus on graph vs. vector** (per
[GraphRAG-Bench, ICLR'26](https://github.com/GraphRAG-Bench/GraphRAG-Benchmark)):
graph wins when (a) coherent domain corpus, (b) questions require
synthesizing across many documents, (c) indexing pass is affordable.
Vector wins on point-fact lookup, noisy/heterogeneous corpora, cost,
latency.

**Recommendation:** for a self-hosted small-model setup with a coherent
domain corpus, LightRAG is the production try-this. Skip full Microsoft
GraphRAG (cost). Watch for LazyGraphRAG GA and swap when shipped.

---

## 3. Long context did not replace RAG

The 2026 reality killed the "just stuff the corpus" dream.

- **Claude Opus 4.7** regressed on multi-needle 1M-context retrieval —
  32.2% on MRCR v2 8-needle at 1M vs. 78.3% for Opus 4.6.
  ([source](https://blog.wentuo.ai/en/claude-opus-4-7-long-context-regression-en.html))
- Multi-needle leaderboard at 1M tokens: Gemini 3 leads at 89%,
  GPT-5.5 at 74%, Opus 4.7 at 56%.
  ([source](https://www.digitalapplied.com/blog/gpt-5-5-vs-claude-opus-4-7-frontier-comparison))
- **RAGAS 2.0 (April 2026) study** found faithfulness dropped up to **40%**
  when context windows were overloaded with irrelevant filler.
  ([source](https://open-techstack.com/blog/rag-vs-long-context-2026/))

**Effective context for multi-needle workloads is 200–400K** for current
frontier models even when the marketed window is 1M+. Above that, attention
dilutes, "lost in the middle" kicks in, latency explodes, cost is linear.

**Gemma 4 specifically: 128K context.** Plenty for the retrieve-then-stuff
pattern, but cap effective working context at ~64K to stay above the
lost-in-the-middle floor for 8–31B models. Not a corpus replacement.

**The winning shape:** RAG retrieves the top 50–200K most relevant tokens
→ long-context model reasons carefully. "Just dump everything" is the
niche choice for single long-document analysis (legal contract, research
paper).

---

## 4. Memory architectures

Real movement in 2025–2026. Five-way taxonomy
([SurePrompts](https://sureprompts.com/blog/agent-memory-architectures-compared-2026)):
provider-managed, self-managing (Letta), CRUD memory layer (mem0), vector
RAG, custom in-app schema.

| Project | Camp | Maturity | License | Best for |
|---------|------|----------|---------|----------|
| [Letta](https://github.com/letta-ai/letta) (formerly MemGPT) | Agent-edited state. **Feb 2026: Context Repositories ("MemFS")** — memory projected into git-backed files operated on via bash/computer-use tools | Production | Apache 2.0 | Long-horizon coherence as the product, strong models only |
| [mem0](https://mem0.ai/research) | CRUD memory layer, vector + optional graph, three-level hierarchy | Production | Apache 2.0 | "Remember the user across sessions" consumer-app cases |
| Zep | Graph-based temporal memory, claims 75.14% on LoCoMo (contested vs mem0) | Production | Commercial + community OSS | Temporal recall + graph relationships |
| [A-MEM](https://github.com/agiresearch/a-mem) (arxiv 2502.12110) | Zettelkasten-inspired — adding new memories triggers updates to existing ones, evolving graph | Experimental | Research | If you want to play with self-organizing memory graphs |
| Hindsight, Memvid, Supermemory, MemPalace | Various | Experimental | Various | Watch, don't commit |

**For Gemma-4-sized models specifically:** lean toward mem0-style managed
memory. Letta's self-edit pattern needs a model that reliably calls the
memory-save tool every time it should — Gemma 4 won't. The benchmarking
fight between mem0 and Zep on LoCoMo is contested; pick on integration
fit, not on the leaderboard.

---

## 5. Retrieval-augmented planning and multi-hop

The 2023-era patterns (ReAct, IRCoT, Self-Ask, DRAGIN) remain the
conceptual foundation. Known weaknesses: error propagation, noisy evidence
sets, fixed step budgets.

**Practical pattern that converged in 2026:**
```
ReAct loop + cross-encoder reranker on every retrieval
         + explicit "do I have enough?" verification step before answering
```

The verification step is the real change from 2023. Agents are expected
to check their evidence rather than just synthesize.

**Recent academic:**
- [PRISM (arxiv 2510.14278)](https://arxiv.org/pdf/2510.14278) and
  [RELOOP (arxiv 2510.20505)](https://arxiv.org/pdf/2510.20505) — late
  2025 frameworks attempting to fix the precision-recall imbalance of
  IRCoT-style approaches. Experimental.
- [Four-axis design framework survey (arxiv 2601.00536)](https://arxiv.org/html/2601.00536v1)
  — useful 2026 organization of the design space.

---

## 6. Deep-research agents

Heavy activity. OpenAI Deep Research (early 2025) set the template;
open-source caught up fast.

| Project | Maturity | License | Notes |
|---------|----------|---------|-------|
| OpenAI / Google Deep Research | Closed | — | Reference target. OpenAI ~67% GAIA validation |
| [smolagents/open_deep_research](https://github.com/huggingface/smolagents/tree/main/examples/open_deep_research) | Production code, experimental quality with small models | Apache 2.0 | 55% GAIA validation with GPT-4o. CodeAgent uses ~30% fewer steps than ToolCallingAgent |
| [gpt-researcher](https://github.com/assafelovic/gpt-researcher) | Production | Apache 2.0 | Most mature open option. Planner + search + reader + report writer with citations |
| [LangChain open_deep_research](https://github.com/langchain-ai/open_deep_research) | Production | MIT | Tavily search default, MCP support, model-agnostic |
| Perplexity Sonar API | Production, closed | — | 37% citation hallucination rate (vs ChatGPT Search 67%, Grok 3 94%) per Columbia Journalism Review audit. Best closed-source citation grounding |

**For Gemma 4 specifically:** 7B–13B class models work in deep-research
frameworks but quality drops noticeably vs. GPT-4o-class. **The 31B
variant is probably the sweet spot for self-hosted deep research;
smaller variants struggle with the planning step.**

---

## 7. Hybrid / late-interaction retrievers — alive and well

"Dense + reranker won" is **not** the 2026 consensus.

| Approach | Maturity | License | When |
|----------|----------|---------|------|
| **ColBERT v2** late-interaction, per-token embeddings | Production | Apache 2.0 | Retrieval-heavy reference architecture |
| [**ColPali**](https://arxiv.org/html/2407.01449v5) — late-interaction over document images via VLMs, OCR-free | Beta → production transition | Apache 2.0 | Visually rich documents (tables, charts, forms). Storage cost is the gotcha (multi-vector per page). Zilliz/Milvus and Weaviate ship integrations |
| **SPLADE** — learned sparse, lexical + semantic | Production | Apache 2.0 | Hybrid pipelines. Lexical signal load-bearing for technical corpora |
| [**BGE-M3**](https://bge-model.com/bge/bge_m3.html) — dense + sparse + multi-vector in one model | Production | MIT | Multilingual default |
| [**Qwen3-Embedding**](https://github.com/QwenLM/Qwen3-Embedding) (0.6B/4B/8B) — 8B topped MTEB multilingual at 70.58 mid-2025 | Production | Apache 2.0 | Strong open default, especially multilingual |

**Settled production pattern**
([source](https://tianpan.co/blog/2026-04-12-hybrid-search-production-bm25-dense-embeddings)):

```
BM25 + dense ANN (parallel) → RRF fusion → top-100 candidates
                                        → cross-encoder rerank → top-5 → LLM
```

BM25 stubbornly wins on literal matches (error codes, product SKUs,
identifiers) — embeddings underperform on those. "Embeddings alone" has
been mostly retired in serious systems.

---

## 8. Reranking and verification

Cross-encoder reranking is table stakes. Choice of model:

| Reranker | License | Notes |
|----------|---------|-------|
| [BGE-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) | MIT | Multilingual, self-hostable, price/performance default |
| Cohere Rerank v3.5 | Closed/paid | 4096-token context, JSON / semi-structured. "Just works" choice |
| [Jina Reranker v3](https://jina.ai/models/jina-reranker-v3/) | Apache 2.0 | 0.6B listwise (query + all candidates in one window). 81.33% Hit@1 at 188ms |
| Qwen3 Reranker (0.6B/4B/8B) | Apache 2.0 | Competitive with BGE/Jina |
| Voyage rerank-2.5, mixedbread mxbai, nemotron reranker | Various | Also in the mix. Nemotron at 83% Hit@1 / 243ms currently the accuracy leader per AIMultiple benchmark |

**Verification patterns (genuinely new in 2026):**
- Explicit "is this passage relevant?" LLM call after rerank
- Citation-supported answers — every claim must point to a retrieved
  passage
- Groundedness scores via [Ragas](https://github.com/explodinggradients/ragas)

**Ragas production thresholds**: faithfulness ≥ 0.9, answer relevancy
≥ 0.85, context precision ≥ 0.8.

---

## 9. Production-vs-experimental quick reference

| Thing | Maturity | License | When to use |
|-------|----------|---------|-------------|
| Hybrid BM25 + dense + RRF + cross-encoder rerank | **Production**, settled | varies | Default starting point for any new RAG |
| BGE-reranker-v2-m3 / Jina v3 / Qwen3-Reranker | **Production** | MIT/Apache | The reranker step above |
| LangGraph / LlamaIndex Workflows | **Production** | MIT | Implementing CRAG / Self-RAG–shaped flows |
| gpt-researcher / smolagents open_deep_research | **Production** (open) | Apache/MIT | Deep-research agents |
| LightRAG | **Production** for SMB corpora | MIT | Graph-aware retrieval without MSR cost |
| LazyGraphRAG | **Beta** (Q1–Q2 2026 GA) | MIT | Replacing full GraphRAG when shipped |
| mem0 | **Production** | Apache 2.0 | "Remember the user" memory |
| Letta + Context Repositories (MemFS) | **Beta–Production** | Apache 2.0 | Long-horizon agent state, strong models only |
| ColPali | **Beta–production** | Apache 2.0 | Visually rich documents, accept storage cost |
| Long context as RAG-replacement | **Niche** | n/a | Single long document, not corpus |
| Self-RAG original models, FLARE original repo | **Paper-only** | research | Read the paper, re-implement the pattern |
| A-MEM | **Experimental** | research | Self-organizing memory graph experiments |
| PRISM / RELOOP | **Experimental** | research | Multi-hop precision experiments |

**Genuine convergence:**
- Hybrid retrieval + cross-encoder rerank is the universal base.
- CRAG-shaped self-correction is the dominant agentic RAG pattern.
- Long context complements RAG; doesn't replace it.
- Citations / grounded answers are expected.

**Still no consensus on:**
- Which agent framework wins (LangGraph vs LlamaIndex Workflows vs
  smolagents vs roll-your-own).
- Memory: managed CRUD layer vs agent-edited state.
- When graph beats vector for a given corpus (domain-dependent).
- Whether tool-call agents or code-writing agents are the better
  substrate (smolagents data favors code; production deployments split).
- Memory benchmarking — LoCoMo numbers actively contested.

---

## 10. Picking a stack for a Gemma 4 agent

For a self-hosted Gemma 4 26B/31B agent that already does basic
tool-calling + embedding RAG and wants to step up:

1. **Base retrieval**: hybrid BM25 + dense (Qwen3-Embedding-4B or
   BGE-M3) with RRF fusion, then BGE-reranker-v2-m3 cross-encoder.
   Non-experimental but the foundation everything else assumes.
2. **Agentic layer**: CRAG-shaped flow in LangGraph — evaluate retrieved
   chunks, fall back to web search if score is low, verify before
   answering. Skip Self-RAG-the-models (won't work with Gemma); use
   the pattern.
3. **Memory**: mem0 for user-facing recall. Skip Letta unless the agent
   specifically needs long-horizon autonomous behavior — Gemma 4 will
   sometimes drop tool calls and you'll lose memory silently if the
   model owns it.
4. **Web search**: gpt-researcher or smolagents/open_deep_research,
   with Gemma 4 31B doing the planning/synthesis. Expect 30–40% lower
   quality than GPT-4o-class; budget for it.
5. **Graph experiment**: LightRAG on whichever surface has a coherent
   domain corpus. Skip full MSR GraphRAG (cost). Watch for LazyGraphRAG
   GA and swap.
6. **Long context**: don't lean on it. 128K Gemma 4 context is for
   "retrieve a lot then reason," not "skip retrieval." Cap effective
   working context at ~64K.
7. **Worth a one-off experiment**: ColPali on any visual-document
   corpus (genuinely different OCR-free path), A-MEM for memory if
   mem0 feels too static, PRISM-style multi-hop if you have a benchmark
   question your CRAG flow can't handle.

**Don't spend time on**: original 2023 implementations (Self-RAG models,
FLARE repo, MemGPT pre-Letta). The ideas survived; the codebases did
not.