How a graph cuts LLM token costs
Context graphs · 5 min read
Most LLM cost problems are context problems. If every query ships the whole knowledge base into the prompt, your per-query cost grows with everything you've ever learned. Retrieval from a context graph inverts that: the query bounds a traversal, the traversal returns a neighbourhood, and the neighbourhood stays roughly the same size no matter how large the graph gets.
The context-stuffing tax
The simplest way to ground a model is to paste in everything it might need. It's also a tax that compounds: input tokens scale with corpus size, you pay it on every call, and long contexts degrade recall — the model attends worst exactly when you've paid the most. Caching helps with repeated prefixes, but the fundamental coupling remains: cost tracks what you know, not what the question needs.
Retrieval bounded by neighbourhood, not corpus
A context graph breaks that coupling. Entities and facts are stored connected, so answering a query means resolving its entities and walking their neighbourhood — the facts about them, the sources, the one-hop context. The size of that result is set by the traversal (LIMIT, hop depth, recency filters), not by the size of the graph. A 10× bigger knowledge base makes the walk no wider.
The measured numbers
On the public entity-graph benchmark at 2,000 entities, sending the knowledge base into context costs 202,285 tokens per query. Retrieving the relevant subgraph from the context graph instead costs 2,668 — about 199,617 tokens avoided per query, a ~98.7% reduction. At $3 per million input tokens, that's roughly $599 saved per thousand queries. Token counts were measured with the GPT-4 BPE tokenizer (cl100k_base).
The honest caveat: the exact ratio depends on your corpus and your traversals. The structural claim is the durable one — baseline tokens scale with corpus size, graph-retrieval tokens scale with neighbourhood size.
The gap widens as you grow
At small scale, stuffing works fine — below a few hundred entities the whole knowledge base fits comfortably in context and a graph is overhead. The crossover comes as the corpus grows: baseline cost climbs linearly while retrieval cost stays flat, so the percentage saved keeps rising. Teams usually feel this as a bill that doubled while traffic didn't.
Doing this with CognoDB
CognoDB is the retrieval layer: model entities and facts as a graph, then let each query traverse only what it needs — over Bolt with Cypher from your application, or directly from the agent through the built-in MCP server. Full-text BM25 search resolves fuzzy entity mentions to entry-point nodes inside the database. Traversals return in milliseconds, so the token savings don't come at a latency cost.
MATCH (e:Entity)<-[:ABOUT]-(f:Fact)
WHERE e.name IN $entities AND f.observed_at > $since
RETURN e.name, f.statement, f.observed_at
ORDER BY f.observed_at DESC
LIMIT 25Questions
Common questions.
How much does graph retrieval actually save?
On the public entity-graph benchmark at 2,000 entities: 202,285 tokens per query for context-stuffing versus 2,668 with graph retrieval — about 98.7% fewer, or roughly $599 per thousand queries at $3 per million input tokens. Your ratio depends on corpus size and traversal shape; the saving grows as the knowledge base does.
Doesn't prompt caching solve this already?
Caching discounts repeated prefixes; it doesn't change what you send. A cached 200K-token context is still slower, still degrades recall, and still couples cost to corpus size. Retrieval reduces what enters the prompt in the first place, and the two techniques stack.
Is this the same as RAG?
It's the graph flavour of it. Classic vector RAG retrieves similar text chunks; graph retrieval traverses relationships from the entities in the query, which keeps results bounded and attaches provenance. Many systems use both — see our GraphRAG guide.
When is a context graph not worth it?
When the knowledge base is small enough to fit in context comfortably — below a few hundred entities the graph is premature. The signal to adopt one is input-token spend growing with your corpus rather than your traffic.
Keep reading
Context graphs
What is a context graph?
A context graph is the connected, queryable record of what an AI system currently knows: entities, facts, sources and time. How it differs from a knowledge graph, and how agents read and write one.
ReadContext graphs
AI agent memory
Persistent memory for AI agents, built as a graph: why transcripts don't scale, the entity–fact–source schema that does, the write and read paths in Cypher, and how multi-agent memory stays sane.
ReadGraphRAG
Graph RAG vs vector RAG
Vector RAG retrieves by similarity; graph RAG retrieves by traversing relationships. A side-by-side comparison, where each wins, and why the 2026 answer is usually a hybrid of the two.
ReadStart now
~98.7%
token efficiency at 2,000 entities (see the footnotes above)
Try the ideas on a real graph.
A free instance takes about a minute and no card. Every Cypher snippet on this page runs against it unchanged.
First-graph path
LiveCreate a free instance
No card. Ready in about a minute.
Connect your driver
bolt+s:// URI into the driver you already use.
Write two MERGEs
That's the entire shape of agent memory.
Point an agent at it
One MCP config block. No integration code.