docs: May 2026 tool-calling update + agentic retrieval landscape
Two new corpus docs plus targeted GOTCHAS corrections based on Ollama/llama.cpp/vLLM issue tracker review, late-2025/early-2026 research papers, and convergent empirical evidence from two production Gemma 4 agents. New: - CORPUS_tool_calling_2026-05.md: current state of tool calling. Properly characterizes the think flag (parser-side bug, not model-side; recipe by loop shape), surfaces ollama/ollama#15539, documents the "doesn't fire tools without explicit ask" research (Probe & Prefill, BiasBusters, When2Call), XGrammar-2. - CORPUS_agentic_retrieval.md: post-RAG landscape. Hybrid retrieval + cross-encoder rerank as the settled base, CRAG-shaped flows, LightRAG/LazyGraphRAG, mem0 vs Letta MemFS, deep-research agents. Production-vs-experimental sorting for self-hosted small-model use. Updates: - GOTCHAS.md: the think:false rule, Vulkan unused-token loop, and Ollama 0.20 streaming bug all marked SUPERSEDED/FIXED/PARTIALLY FIXED in-place with pointers to the May update. - README.md: indexed the two new docs. - .gitignore: added private/ for bot-specific notes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Mortdecai
@@ -0,0 +1,308 @@
|
||||
# Agentic Information Retrieval Beyond RAG, May 2026
|
||||
|
||||
> A view of where retrieval and memory architectures are converging for
|
||||
> agentic applications, with explicit production-vs-experimental sorting
|
||||
> for self-hosted small-model deployments (Gemma 4 26B–31B, Qwen3 8–14B).
|
||||
>
|
||||
> Last updated: 2026-05-25. Cuts off long-context-as-RAG-replacement,
|
||||
> graph retrieval, agent memory, and deep-research-agent patterns.
|
||||
|
||||
## Headlines
|
||||
|
||||
1. **The 2023-era named patterns (Self-RAG, CRAG, FLARE, MemGPT) survived
|
||||
as design templates but their original codebases are mostly stale.**
|
||||
The 2026 production stack reimplements them as graph nodes inside
|
||||
LangGraph or LlamaIndex Workflows. Use the patterns; don't try to use
|
||||
the original repos.
|
||||
2. **Long context did not replace RAG.** Multi-needle retrieval at 1M
|
||||
tokens regressed in some frontier models in 2026; even where it works,
|
||||
the cost/latency math means RAG-then-stuff stays the winning shape.
|
||||
The "just dump the corpus" approach is now a niche choice for
|
||||
single-document deep reasoning, not corpus retrieval.
|
||||
3. **GraphRAG got cost-viable.** Microsoft's
|
||||
[LazyGraphRAG](https://www.microsoft.com/en-us/research/blog/lazygraphrag-setting-a-new-standard-for-quality-and-cost/)
|
||||
matches or beats full GraphRAG quality at indexing cost equal to vector
|
||||
RAG and query cost 0.1% of full GraphRAG. GA targeting Q1–Q2 2026 in
|
||||
the MSR repo. For SMB corpora today, [LightRAG](https://github.com/hkuds/lightrag)
|
||||
is the production option.
|
||||
4. **Memory architectures split into two camps**: managed CRUD layer
|
||||
(mem0, Zep) vs. agent-edited state (Letta with Context Repositories
|
||||
"MemFS", A-MEM). For small open models, lean toward managed — Gemma 4
|
||||
will sometimes drop tool calls and you'll lose memory silently if the
|
||||
model owns it.
|
||||
5. **The settled retrieval base layer is hybrid BM25 + dense + RRF
|
||||
fusion + cross-encoder rerank.** "Dense embeddings alone" has been
|
||||
retired in serious systems. The lexical signal is load-bearing for
|
||||
error codes, identifiers, and technical corpora that embeddings
|
||||
underperform on.
|
||||
|
||||
---
|
||||
|
||||
## 1. Agentic RAG — patterns, not libraries
|
||||
|
||||
The original papers all date from 2023. In 2026 they exist as patterns
|
||||
absorbed into agent frameworks.
|
||||
|
||||
| Pattern | Origin | What survived | What to use |
|
||||
|---------|--------|---------------|-------------|
|
||||
| **Self-RAG** | Asai et al., ICLR 2024 | Reflection-token idea (model decides per-step whether to retrieve, critique chunks) | Re-implement as graph nodes; the trained 7B/13B models aren't being updated |
|
||||
| **CRAG (Corrective RAG)** | Yan et al., 2024 | retrieve → evaluate → correct → generate | LangGraph cookbook implementation is now the canonical artifact |
|
||||
| **FLARE** | Jiang et al., 2023 | Uncertainty-triggered retrieval (low-confidence span ⇒ re-retrieve) | Pattern survives, original repo stale |
|
||||
| **PRISM / RELOOP** | arxiv 2510.14278, 2510.20505 (late 2025) | New, multi-hop with precision-focused control flow | Experimental — track if multi-hop precision is your bottleneck |
|
||||
|
||||
**Frameworks ranked by fit for branching agentic RAG flows:**
|
||||
|
||||
- **LangGraph** (graph-native, best for CRAG/Self-RAG-shaped branching)
|
||||
- **LlamaIndex Workflows** (event-driven, similar story)
|
||||
- **Haystack** (more structured pipelines, weaker on dynamic branching)
|
||||
|
||||
GitHub star order, January 2026:
|
||||
LangChain 125K · Dify 114K · RAGFlow 70K · LlamaIndex 46.5K.
|
||||
([source](https://florinelchis.medium.com/top-10-rag-frameworks-on-github-by-stars-january-2026-e6edff1e0d91))
|
||||
|
||||
**Convergence:** yes, on the pattern — agent decides when to retrieve,
|
||||
evaluates retrieved chunks, optionally re-queries. **No convergence** on
|
||||
which framework owns it.
|
||||
|
||||
---
|
||||
|
||||
## 2. GraphRAG: the cost story
|
||||
|
||||
Microsoft's original GraphRAG had a ~$33K indexing bill on large corpora,
|
||||
making it a non-starter outside well-funded enterprises. 2026 changed
|
||||
that.
|
||||
|
||||
| Project | Maturity | License | When to use |
|
||||
|---------|----------|---------|-------------|
|
||||
| [Microsoft GraphRAG](https://github.com/microsoft/graphrag) | Production but expensive | MIT | Global-question answering over stable corpora when budget is unconstrained |
|
||||
| [LazyGraphRAG](https://www.microsoft.com/en-us/research/blog/lazygraphrag-setting-a-new-standard-for-quality-and-cost/) | Beta, GA Q1–Q2 2026 | MIT | The new default — same quality, 0.1% query cost |
|
||||
| [LightRAG](https://github.com/hkuds/lightrag) (EMNLP 2025) | Production | MIT | SMB corpora today, Docker-deployable, PG+pgvector+AGE one-DB path |
|
||||
| [nano-graphrag](https://github.com/gusye1234/nano-graphrag), [fast-graphrag](https://github.com/circlemind-ai/fast-graphrag) | Experimental | MIT | Hackable baselines, sub-1M-doc cases |
|
||||
|
||||
**2026 consensus on graph vs. vector** (per
|
||||
[GraphRAG-Bench, ICLR'26](https://github.com/GraphRAG-Bench/GraphRAG-Benchmark)):
|
||||
graph wins when (a) coherent domain corpus, (b) questions require
|
||||
synthesizing across many documents, (c) indexing pass is affordable.
|
||||
Vector wins on point-fact lookup, noisy/heterogeneous corpora, cost,
|
||||
latency.
|
||||
|
||||
**Recommendation:** for a self-hosted small-model setup with a coherent
|
||||
domain corpus, LightRAG is the production try-this. Skip full Microsoft
|
||||
GraphRAG (cost). Watch for LazyGraphRAG GA and swap when shipped.
|
||||
|
||||
---
|
||||
|
||||
## 3. Long context did not replace RAG
|
||||
|
||||
The 2026 reality killed the "just stuff the corpus" dream.
|
||||
|
||||
- **Claude Opus 4.7** regressed on multi-needle 1M-context retrieval —
|
||||
32.2% on MRCR v2 8-needle at 1M vs. 78.3% for Opus 4.6.
|
||||
([source](https://blog.wentuo.ai/en/claude-opus-4-7-long-context-regression-en.html))
|
||||
- Multi-needle leaderboard at 1M tokens: Gemini 3 leads at 89%,
|
||||
GPT-5.5 at 74%, Opus 4.7 at 56%.
|
||||
([source](https://www.digitalapplied.com/blog/gpt-5-5-vs-claude-opus-4-7-frontier-comparison))
|
||||
- **RAGAS 2.0 (April 2026) study** found faithfulness dropped up to **40%**
|
||||
when context windows were overloaded with irrelevant filler.
|
||||
([source](https://open-techstack.com/blog/rag-vs-long-context-2026/))
|
||||
|
||||
**Effective context for multi-needle workloads is 200–400K** for current
|
||||
frontier models even when the marketed window is 1M+. Above that, attention
|
||||
dilutes, "lost in the middle" kicks in, latency explodes, cost is linear.
|
||||
|
||||
**Gemma 4 specifically: 128K context.** Plenty for the retrieve-then-stuff
|
||||
pattern, but cap effective working context at ~64K to stay above the
|
||||
lost-in-the-middle floor for 8–31B models. Not a corpus replacement.
|
||||
|
||||
**The winning shape:** RAG retrieves the top 50–200K most relevant tokens
|
||||
→ long-context model reasons carefully. "Just dump everything" is the
|
||||
niche choice for single long-document analysis (legal contract, research
|
||||
paper).
|
||||
|
||||
---
|
||||
|
||||
## 4. Memory architectures
|
||||
|
||||
Real movement in 2025–2026. Five-way taxonomy
|
||||
([SurePrompts](https://sureprompts.com/blog/agent-memory-architectures-compared-2026)):
|
||||
provider-managed, self-managing (Letta), CRUD memory layer (mem0), vector
|
||||
RAG, custom in-app schema.
|
||||
|
||||
| Project | Camp | Maturity | License | Best for |
|
||||
|---------|------|----------|---------|----------|
|
||||
| [Letta](https://github.com/letta-ai/letta) (formerly MemGPT) | Agent-edited state. **Feb 2026: Context Repositories ("MemFS")** — memory projected into git-backed files operated on via bash/computer-use tools | Production | Apache 2.0 | Long-horizon coherence as the product, strong models only |
|
||||
| [mem0](https://mem0.ai/research) | CRUD memory layer, vector + optional graph, three-level hierarchy | Production | Apache 2.0 | "Remember the user across sessions" consumer-app cases |
|
||||
| Zep | Graph-based temporal memory, claims 75.14% on LoCoMo (contested vs mem0) | Production | Commercial + community OSS | Temporal recall + graph relationships |
|
||||
| [A-MEM](https://github.com/agiresearch/a-mem) (arxiv 2502.12110) | Zettelkasten-inspired — adding new memories triggers updates to existing ones, evolving graph | Experimental | Research | If you want to play with self-organizing memory graphs |
|
||||
| Hindsight, Memvid, Supermemory, MemPalace | Various | Experimental | Various | Watch, don't commit |
|
||||
|
||||
**For Gemma-4-sized models specifically:** lean toward mem0-style managed
|
||||
memory. Letta's self-edit pattern needs a model that reliably calls the
|
||||
memory-save tool every time it should — Gemma 4 won't. The benchmarking
|
||||
fight between mem0 and Zep on LoCoMo is contested; pick on integration
|
||||
fit, not on the leaderboard.
|
||||
|
||||
---
|
||||
|
||||
## 5. Retrieval-augmented planning and multi-hop
|
||||
|
||||
The 2023-era patterns (ReAct, IRCoT, Self-Ask, DRAGIN) remain the
|
||||
conceptual foundation. Known weaknesses: error propagation, noisy evidence
|
||||
sets, fixed step budgets.
|
||||
|
||||
**Practical pattern that converged in 2026:**
|
||||
```
|
||||
ReAct loop + cross-encoder reranker on every retrieval
|
||||
+ explicit "do I have enough?" verification step before answering
|
||||
```
|
||||
|
||||
The verification step is the real change from 2023. Agents are expected
|
||||
to check their evidence rather than just synthesize.
|
||||
|
||||
**Recent academic:**
|
||||
- [PRISM (arxiv 2510.14278)](https://arxiv.org/pdf/2510.14278) and
|
||||
[RELOOP (arxiv 2510.20505)](https://arxiv.org/pdf/2510.20505) — late
|
||||
2025 frameworks attempting to fix the precision-recall imbalance of
|
||||
IRCoT-style approaches. Experimental.
|
||||
- [Four-axis design framework survey (arxiv 2601.00536)](https://arxiv.org/html/2601.00536v1)
|
||||
— useful 2026 organization of the design space.
|
||||
|
||||
---
|
||||
|
||||
## 6. Deep-research agents
|
||||
|
||||
Heavy activity. OpenAI Deep Research (early 2025) set the template;
|
||||
open-source caught up fast.
|
||||
|
||||
| Project | Maturity | License | Notes |
|
||||
|---------|----------|---------|-------|
|
||||
| OpenAI / Google Deep Research | Closed | — | Reference target. OpenAI ~67% GAIA validation |
|
||||
| [smolagents/open_deep_research](https://github.com/huggingface/smolagents/tree/main/examples/open_deep_research) | Production code, experimental quality with small models | Apache 2.0 | 55% GAIA validation with GPT-4o. CodeAgent uses ~30% fewer steps than ToolCallingAgent |
|
||||
| [gpt-researcher](https://github.com/assafelovic/gpt-researcher) | Production | Apache 2.0 | Most mature open option. Planner + search + reader + report writer with citations |
|
||||
| [LangChain open_deep_research](https://github.com/langchain-ai/open_deep_research) | Production | MIT | Tavily search default, MCP support, model-agnostic |
|
||||
| Perplexity Sonar API | Production, closed | — | 37% citation hallucination rate (vs ChatGPT Search 67%, Grok 3 94%) per Columbia Journalism Review audit. Best closed-source citation grounding |
|
||||
|
||||
**For Gemma 4 specifically:** 7B–13B class models work in deep-research
|
||||
frameworks but quality drops noticeably vs. GPT-4o-class. **The 31B
|
||||
variant is probably the sweet spot for self-hosted deep research;
|
||||
smaller variants struggle with the planning step.**
|
||||
|
||||
---
|
||||
|
||||
## 7. Hybrid / late-interaction retrievers — alive and well
|
||||
|
||||
"Dense + reranker won" is **not** the 2026 consensus.
|
||||
|
||||
| Approach | Maturity | License | When |
|
||||
|----------|----------|---------|------|
|
||||
| **ColBERT v2** late-interaction, per-token embeddings | Production | Apache 2.0 | Retrieval-heavy reference architecture |
|
||||
| [**ColPali**](https://arxiv.org/html/2407.01449v5) — late-interaction over document images via VLMs, OCR-free | Beta → production transition | Apache 2.0 | Visually rich documents (tables, charts, forms). Storage cost is the gotcha (multi-vector per page). Zilliz/Milvus and Weaviate ship integrations |
|
||||
| **SPLADE** — learned sparse, lexical + semantic | Production | Apache 2.0 | Hybrid pipelines. Lexical signal load-bearing for technical corpora |
|
||||
| [**BGE-M3**](https://bge-model.com/bge/bge_m3.html) — dense + sparse + multi-vector in one model | Production | MIT | Multilingual default |
|
||||
| [**Qwen3-Embedding**](https://github.com/QwenLM/Qwen3-Embedding) (0.6B/4B/8B) — 8B topped MTEB multilingual at 70.58 mid-2025 | Production | Apache 2.0 | Strong open default, especially multilingual |
|
||||
|
||||
**Settled production pattern**
|
||||
([source](https://tianpan.co/blog/2026-04-12-hybrid-search-production-bm25-dense-embeddings)):
|
||||
|
||||
```
|
||||
BM25 + dense ANN (parallel) → RRF fusion → top-100 candidates
|
||||
→ cross-encoder rerank → top-5 → LLM
|
||||
```
|
||||
|
||||
BM25 stubbornly wins on literal matches (error codes, product SKUs,
|
||||
identifiers) — embeddings underperform on those. "Embeddings alone" has
|
||||
been mostly retired in serious systems.
|
||||
|
||||
---
|
||||
|
||||
## 8. Reranking and verification
|
||||
|
||||
Cross-encoder reranking is table stakes. Choice of model:
|
||||
|
||||
| Reranker | License | Notes |
|
||||
|----------|---------|-------|
|
||||
| [BGE-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) | MIT | Multilingual, self-hostable, price/performance default |
|
||||
| Cohere Rerank v3.5 | Closed/paid | 4096-token context, JSON / semi-structured. "Just works" choice |
|
||||
| [Jina Reranker v3](https://jina.ai/models/jina-reranker-v3/) | Apache 2.0 | 0.6B listwise (query + all candidates in one window). 81.33% Hit@1 at 188ms |
|
||||
| Qwen3 Reranker (0.6B/4B/8B) | Apache 2.0 | Competitive with BGE/Jina |
|
||||
| Voyage rerank-2.5, mixedbread mxbai, nemotron reranker | Various | Also in the mix. Nemotron at 83% Hit@1 / 243ms currently the accuracy leader per AIMultiple benchmark |
|
||||
|
||||
**Verification patterns (genuinely new in 2026):**
|
||||
- Explicit "is this passage relevant?" LLM call after rerank
|
||||
- Citation-supported answers — every claim must point to a retrieved
|
||||
passage
|
||||
- Groundedness scores via [Ragas](https://github.com/explodinggradients/ragas)
|
||||
|
||||
**Ragas production thresholds**: faithfulness ≥ 0.9, answer relevancy
|
||||
≥ 0.85, context precision ≥ 0.8.
|
||||
|
||||
---
|
||||
|
||||
## 9. Production-vs-experimental quick reference
|
||||
|
||||
| Thing | Maturity | License | When to use |
|
||||
|-------|----------|---------|-------------|
|
||||
| Hybrid BM25 + dense + RRF + cross-encoder rerank | **Production**, settled | varies | Default starting point for any new RAG |
|
||||
| BGE-reranker-v2-m3 / Jina v3 / Qwen3-Reranker | **Production** | MIT/Apache | The reranker step above |
|
||||
| LangGraph / LlamaIndex Workflows | **Production** | MIT | Implementing CRAG / Self-RAG–shaped flows |
|
||||
| gpt-researcher / smolagents open_deep_research | **Production** (open) | Apache/MIT | Deep-research agents |
|
||||
| LightRAG | **Production** for SMB corpora | MIT | Graph-aware retrieval without MSR cost |
|
||||
| LazyGraphRAG | **Beta** (Q1–Q2 2026 GA) | MIT | Replacing full GraphRAG when shipped |
|
||||
| mem0 | **Production** | Apache 2.0 | "Remember the user" memory |
|
||||
| Letta + Context Repositories (MemFS) | **Beta–Production** | Apache 2.0 | Long-horizon agent state, strong models only |
|
||||
| ColPali | **Beta–production** | Apache 2.0 | Visually rich documents, accept storage cost |
|
||||
| Long context as RAG-replacement | **Niche** | n/a | Single long document, not corpus |
|
||||
| Self-RAG original models, FLARE original repo | **Paper-only** | research | Read the paper, re-implement the pattern |
|
||||
| A-MEM | **Experimental** | research | Self-organizing memory graph experiments |
|
||||
| PRISM / RELOOP | **Experimental** | research | Multi-hop precision experiments |
|
||||
|
||||
**Genuine convergence:**
|
||||
- Hybrid retrieval + cross-encoder rerank is the universal base.
|
||||
- CRAG-shaped self-correction is the dominant agentic RAG pattern.
|
||||
- Long context complements RAG; doesn't replace it.
|
||||
- Citations / grounded answers are expected.
|
||||
|
||||
**Still no consensus on:**
|
||||
- Which agent framework wins (LangGraph vs LlamaIndex Workflows vs
|
||||
smolagents vs roll-your-own).
|
||||
- Memory: managed CRUD layer vs agent-edited state.
|
||||
- When graph beats vector for a given corpus (domain-dependent).
|
||||
- Whether tool-call agents or code-writing agents are the better
|
||||
substrate (smolagents data favors code; production deployments split).
|
||||
- Memory benchmarking — LoCoMo numbers actively contested.
|
||||
|
||||
---
|
||||
|
||||
## 10. Picking a stack for a Gemma 4 agent
|
||||
|
||||
For a self-hosted Gemma 4 26B/31B agent that already does basic
|
||||
tool-calling + embedding RAG and wants to step up:
|
||||
|
||||
1. **Base retrieval**: hybrid BM25 + dense (Qwen3-Embedding-4B or
|
||||
BGE-M3) with RRF fusion, then BGE-reranker-v2-m3 cross-encoder.
|
||||
Non-experimental but the foundation everything else assumes.
|
||||
2. **Agentic layer**: CRAG-shaped flow in LangGraph — evaluate retrieved
|
||||
chunks, fall back to web search if score is low, verify before
|
||||
answering. Skip Self-RAG-the-models (won't work with Gemma); use
|
||||
the pattern.
|
||||
3. **Memory**: mem0 for user-facing recall. Skip Letta unless the agent
|
||||
specifically needs long-horizon autonomous behavior — Gemma 4 will
|
||||
sometimes drop tool calls and you'll lose memory silently if the
|
||||
model owns it.
|
||||
4. **Web search**: gpt-researcher or smolagents/open_deep_research,
|
||||
with Gemma 4 31B doing the planning/synthesis. Expect 30–40% lower
|
||||
quality than GPT-4o-class; budget for it.
|
||||
5. **Graph experiment**: LightRAG on whichever surface has a coherent
|
||||
domain corpus. Skip full MSR GraphRAG (cost). Watch for LazyGraphRAG
|
||||
GA and swap.
|
||||
6. **Long context**: don't lean on it. 128K Gemma 4 context is for
|
||||
"retrieve a lot then reason," not "skip retrieval." Cap effective
|
||||
working context at ~64K.
|
||||
7. **Worth a one-off experiment**: ColPali on any visual-document
|
||||
corpus (genuinely different OCR-free path), A-MEM for memory if
|
||||
mem0 feels too static, PRISM-style multi-hop if you have a benchmark
|
||||
question your CRAG flow can't handle.
|
||||
|
||||
**Don't spend time on**: original 2023 implementations (Self-RAG models,
|
||||
FLARE repo, MemGPT pre-Letta). The ideas survived; the codebases did
|
||||
not.
|
||||
Reference in New Issue
Block a user