Vector RAG Isn’t Enough — I Built a Context Graph Layer for Multi-Agent Memory
Vector RAG Isn’t Enough — I Built a Context Graph Layer for Multi-Agent Memory
Publish Date: 2026-06-25 14:37:00
Source Domain: towardsdatascience.com
- I wasn’t trying to build a new memory architecture. I was trying to understand why one agent kept forgetting decisions made by another. The benchmark came later.
- Multi-agent systems lose cross-agent decisions because flat transcripts and vector search both have a structural blind spot — not just a noise problem.
- A context graph stores facts as entities and relationships instead of text chunks, so it can answer questions that need two facts combined.
- This is not a concept. Three memory architectures, five scripted scenarios, 18 graded queries, fully deterministic, zero LLM calls.
- Context graph: 88.9% accuracy at 26.9 tokens/query. Raw history dump: 61.1% accuracy at 490.9 tokens/query. Vector-only RAG: 50.0% accuracy at 75.9 tokens/query.
- I found two real bugs building this — stale-fact retrieval and an entity-matching gap. Both are in the article.
The Problem That Made Me Build This
I built a three-agent pipeline that worked great for short tasks. But the moment the conversation dragged on and an agent needed to recall a past decision, the whole thing fell apart.
Here is exactly how it broke: Agent_Planner would decide the project should use PostgreSQL. Then, twenty turns of “sounds good” and “I’ll get to it” would pass. Eventually, Agent_Reviewer would pipe up and ask what storage technology we were using. Even with the entire raw transcript sitting right there in the context window, the agent couldn’t answer reliably.
I was running this pipeline locally as a side project for EmiTechLogic just to see how far I could push multi-agent coordination before it hit a wall. Turns out, it didn’t take very long.
Initially, I assumed this was just a model limitation. It isn’t. It is a memory architecture problem that usually triggers one of two massive headaches depending on how you try to fix it.
The Alternative Fix: Vector Search and the Relational Trap
If you switch to vector search, you fix the noise…
