Introduction
Before jumping into retrieval, I wanted a memory layer that behaved less like a vector database and more like the way humans connect ideas. So there is this idea to rather construct a semi-graph memory layer instead of storing isolated chunks, this layer combines semantic search with explicit relationships between concepts, people, facts, etc. The result is better, richer casuality that is visible in each answer of RAG system. In this post, I’ll walk through the architecture, the design decisions, and the lessons learned while building it.
Traditional RAG systems retrieve documents because they are semantically similar to a query. That works well for factual lookup, but it often misses context. Humans rarely remember isolated facts—we remember relationships: who introduced whom, what caused an event, which ideas evolved from earlier ones, or how two concepts are connected.
The semi-graph memory layer is an attempt to capture exactly that. Instead of treating each chunk as an independent embedding, it enriches memory with explicit links between entities, concepts, events, and observations. Semantic similarity becomes the entry point, while graph traversal provides the surrounding context.
The result is retrieval that is not only relevant but also coherent. Rather than returning a collection of loosely related passages, the system reconstructs a chain of connected knowledge, making answers more complete, more explainable, and closer to the way people naturally reason.
Idea taken from Andrej Karpathy’s post here.