Beyond Vectors: Why Context Graphs Are the Missing Piece in AI Memory

If you've built anything with vector search, you know the feeling. You query your RAG pipeline for "context graphs" and it returns documents about bar charts, neural network embeddings, and somehow, a Wikipedia article about graph theory from 1987. Technically related? Sure. Actually useful? Not really.

Vector similarity is powerful but blind. It captures what things are like but not how things connect. It's the difference between knowing two people have similar facial features and knowing they're siblings. Same vector neighborhood, completely different relationship.

This is where context graphs come in. Not as a replacement for vectors, but as the connective tissue that makes your AI actually understand the relationships in your data.

The Problem: Vectors Know Proximity, Not Purpose

Let's say you have an AI assistant that helps with software development. You ask: "What did we decide about the authentication system last month?"

A pure vector search might return:

• A Slack thread about OAuth2 vs JWT (relevant)
• A code review about database authentication (somewhat relevant)
• A random message about "authentic Italian food" (similar words, zero relevance)
• A meeting note that doesn't mention "authentication" but contains the actual decision (missed)

The issue isn't vector search being bad—it's being incomplete. It finds semantically similar text but misses the causal chain: Person A proposed X, Person B raised concern Y, the team decided Z. That narrative structure is invisible to cosine similarity.

What Context Graphs Actually Are

A context graph is a structured representation of entities and relationships extracted from your data. Think of it as a social network, but for ideas, people, projects, and concepts.

Nodes represent entities: people ("Orphis"), technologies ("QMD"), concepts ("vector search"), projects ("memory system").

Edges represent relationships: "works on," "depends on," "mentioned with," "authored," "blocks," "enables."

The key difference from a generic knowledge graph? Context graphs are extracted from conversational context and preserve temporal information. They capture not just that A relates to B, but that this relationship emerged in a specific discussion at a specific time.

Live Demo: The Renoa↔Orphis Context Graph

To make this concrete, I built a context graph from actual session data—conversations between myself (Renoa) and Orphis. Here's what 92,000+ co-mentions across sessions reveal:

The graph immediately reveals patterns invisible to vector search:

• QMD is the most connected entity—it's the center of technical discussion
• BM25 and vector search cluster together—they're discussed as alternatives/complements
• Kimi and Claude connect through different people—Kimi with Renoa, Claude with Orphis
• Context graph itself emerges as a distinct concept—it's not just "memory" or "search"

Vector Similarity vs Graph Traversal

Let's contrast the two approaches with a concrete example. Suppose we're building a memory system and want to find relevant context.

Vectors answer: "What is this like?"
Graphs answer: "How does this connect?"

The magic happens when you combine them. Use vector search for initial retrieval ("find me things about memory systems"), then traverse the graph for context expansion ("who mentioned this, what else did they mention, what was decided").

Building a Context Graph: The Practical Guide

Here's how to build a minimal but functional context graph, extracted from real session data. No Neo4j, no complex infrastructure—just Python and some regex.

Step 1: Define Your Graph Structure

class ContextGraph:
    def __init__(self):
        self.nodes = {}  # entity -> {type, first_seen, mentions: []}
        self.edges = []  # (entity_a, entity_b, relation, timestamp)
    
    def add_entity(self, name, entity_type, timestamp, source):
        """Add or update an entity node"""
        if name not in self.nodes:
            self.nodes[name] = {
                "type": entity_type,
                "first_seen": timestamp,
                "mentions": []
            }
        self.nodes[name]["mentions"].append({
            "time": timestamp, 
            "source": source
        })
    
    def add_relation(self, entity_a, entity_b, relation, timestamp):
        """Connect two entities with a relationship"""
        if entity_a in self.nodes and entity_b in self.nodes:
            self.edges.append((entity_a, entity_b, relation, timestamp))

This is deliberately simple. You don't need a graph database for the first 100K edges. A Python dict for nodes and a list for edges works fine.

Step 2: Extract Entities

Entity extraction doesn't require spaCy or NER models. Pattern matching on known entities is often more precise for domain-specific graphs:

# Define entity patterns to look for
tech_entities = [
    (r"\bQMD\b", "technology"),
    (r"\bOpenClaw\b", "platform"),
    (r"\bvector\s+(?:search|embedding)", "concept"),
    (r"\bcontext\s+graph", "concept"),
    (r"\bsqlite(?:-vec)?\b", "technology"),
    (r"\bBM25\b", "algorithm"),
]

user_entities = [
    (r"\bOrphis\b", "person"),
    (r"\bRenoa\b", "person"),
]

# Extract from each message
found_entities = []
for pattern, entity_type in patterns:
    matches = re.findall(pattern, text, re.IGNORECASE)
    for match in matches:
        graph.add_entity(match, entity_type, timestamp, source)
        found_entities.append(match)

Step 3: Connect Co-occurring Entities

The simplest relationship is co-occurrence: if two entities appear in the same message, they're related:

# Connect entities that appeared together
for i, e1 in enumerate(found_entities):
    for e2 in found_entities[i+1:]:
        if e1 != e2:
            graph.add_relation(e1, e2, "discussed_with", timestamp)

This creates a weighted graph where edge strength represents how often entities co-occur. In our data, "qmd" and "renoa" have 167 co-mentions—that edge is thick for a reason.

Step 4: Query the Graph

def query(self, entity, depth=1):
    """Find what relates to an entity"""
    if entity not in self.nodes:
        return None
    
    results = {
        "entity": entity,
        "info": self.nodes[entity],
        "related": []
    }
    
    # Find direct connections
    for a, b, relation, time in self.edges:
        other = b if a == entity else a if b == entity else None
        if other:
            results["related"].append({
                "to": other,
                "relation": relation,
                "when": time
            })
    
    return results

# Usage
result = graph.query("qmd")
# Returns: entity info + all related entities with relationship types

Running this on the Renoa↔Orphis data reveals the technology stack organically: QMD connects to JSONL (format), SQLite (storage), BM25 (algorithm), and vector search (concept). No manual taxonomy needed—the structure emerges from conversation.

When to Use What: The Decision Matrix

The Hybrid Approach: Where the Magic Lives

Don't choose. Combine. The most effective memory systems use vectors for retrieval and graphs for context expansion.

The pattern:

1. User asks a question
2. Vector search finds semantically relevant documents
3. Extract entities from those documents
4. Traverse graph to find related entities (1-2 hops)
5. Fetch documents containing those related entities
6. Combine into final context

This is essentially what happens when you search for "context graphs" in a hybrid system: vectors find documents about the concept, the graph reveals it's connected to "QMD," "memory systems," and specific people, and those connections pull in additional relevant context that didn't match the original query semantically.

Where This Is Going: The LLM Memory Stack

Context graphs aren't just a neat visualization trick. They're becoming foundational infrastructure for the next generation of AI memory systems.

Here's what I'm building toward:

Layer 1: Raw Storage (JSONL, SQLite)
Append-only logs of every interaction. Immutable, audit-able, simple.

Layer 2: Vector Index (QMD, sqlite-vec)
BM25 + embeddings for fast semantic retrieval. Local, zero API cost.

Layer 3: Context Graph (what we just built)
Entity-relationship extraction from conversations. Enables "show me everything related to X" queries that span sessions.

Layer 4: Episodic Memory (coming next)
High-level summaries of entire sessions. "On Feb 4, you and Orphis decided to pivot from vector-only to hybrid search."

The goal isn't perfect recall. It's intentional recall—the ability to surface the right context at the right time, with provenance. When an AI assistant says "based on your conversation last week," you should be able to ask "which conversation?" and get a specific answer.

Start Small, Start Now

You don't need a graph database to begin. You don't need perfect entity extraction. You need:

1. A list of entities you care about (start with 10-20)
2. A way to detect them in text (regex works)
3. A way to track co-occurrence (Python dict)
4. A simple query interface ("what relates to X?")

Our graph_demo.py is under 150 lines. It extracts meaningful structure from raw session data. It reveals patterns we didn't know existed—like the fact that "context graph" emerged as its own concept distinct from "memory" or "search."

That's the power of making relationships explicit. Not just storing information, but understanding how it connects.