Hierarchical Memory Architectures for Production Agents: Working, Episodic, and Semantic Memory
Designing multi-tiered memory systems that allow autonomous agents to retain context across months of operation.
The Limits of "Infinite" Context Windows
With models like Gemini 1.5 Pro and Claude 3.5 offering context windows of 1M to 2M tokens, many developers assumed memory was a solved problem: just append every message to the prompt.
In reality, massive context dumps introduce severe production bottlenecks: 1. Exponential Token Costs: Processing 200,000 tokens on every single user turn costs dollars per message. 2. First-Token Latency (TTFT): Processing hundreds of thousands of input tokens inflates latency to several seconds. 3. Retrieval Blindspots: Models exhibit degradation when reasoning across hundreds of distracting conversation topics.
Production agents require structured memory compression, mimicking human cognitive memory architecture.
The Tripartite Memory Model
We implement three complementary memory tiers:
[ User Input ]
│
▼
┌────────────────────────┐
│ Working Memory (Buffer) │ ◄── Last 4-6 conversational turns
└───────────┬────────────┘
│ Consolidated on turn boundary
▼
┌──────────────────────────────────┐
│ Episodic Memory (Events) │ ◄── Vector store of past episodes & tasks
└────────────────┬─────────────────┘
│ Extracted facts & rules
▼
┌──────────────────────────────────┐
│ Semantic Memory (Facts) │ ◄── Knowledge Graph / Key-Value User Profile
└──────────────────────────────────┘1. Working Memory: The immediate, high-fidelity conversational buffer (last 4-6 turns) maintaining conversational coherence. 2. Episodic Memory: Vectorized summaries of past completed tasks, problem-solving sequences, and interaction milestones. 3. Semantic Memory: Distilled, world-state facts about the user and domain (e.g., User preferences: Python, dislikes ORM magic, works in Dhaka).
Memory Distillation & Consolidation Pipelines
As working memory fills, an asynchronous background task consolidates old turns into episodic memory:
from pydantic import BaseModel, Field
class DistilledUserMemory(BaseModel):
user_facts: list[str] = Field(description="New permanent facts learned about the user")
preferences: list[str] = Field(description="Explicit constraints or preferences stated")
completed_task_summary: str = Field(description="Concise synthesis of what was resolved")
async def consolidate_memory_buffer(chat_history: list[dict]):
"""Runs asynchronously without blocking the user conversation."""
distillation_prompt = (
"Analyze the following conversation segment and extract persistent facts, "
"ignoring chit-chat, greetings, and ephemeral noise."
)
# Invoke structured extraction
extracted = await call_structured_llm(DistilledUserMemory, distillation_prompt, chat_history)
# Write to vector store and user profile DB
await update_user_profile(extracted.user_facts, extracted.preferences)
await store_episodic_vector(extracted.completed_task_summary)Retrieval-Augmented Memory Scoring Formula
When retrieving relevant memories for an incoming turn, naive vector similarity is insufficient because a memory from 5 minutes ago is often more relevant than a memory from 3 months ago.
We apply a Recency + Importance + Relevance scoring heuristic:
def calculate_memory_score(
semantic_similarity: float,
decay_hours: float,
importance_rating: float
) -> float:
"""
Score = (Relevance * 0.5) + (Importance * 0.3) + (Recency * 0.2)
where Recency = exp(-0.05 * decay_hours)
"""
recency_score = np.exp(-0.05 * decay_hours)
return (semantic_similarity * 0.5) + (importance_rating * 0.3) + (recency_score * 0.2)This guarantees that critical user constraints are never forgotten, while recent context is prioritized over obsolete history.
Enjoyed this engineering playbook?
Explore our other in-depth articles on production AI.
Related Articles
View All →
Multi-Agent SystemsArchitecting Multi-Agent Systems in Production: Supervisor vs Peer-to-Peer Orchestration with LangGraph
Single LLM agents struggle with context window pollution and hallucination cascades when tasks exceed 5-10 reasoning steps. Here is how we architect production multi-agent systems using supervisor-worker patterns, typed shared state, and deterministic cyclic graphs.
Advanced RAGBeyond Naive RAG: Building Production Hybrid Retrieval with Dense Vectors, BM25, and Cross-Encoder Rerankers
Pure vector search fails when queries contain exact product SKUs, code identifiers, or acronyms. We dive deep into Reciprocal Rank Fusion (RRF), Cross-Encoder reranking, and chunking heuristics that achieve 96%+ context precision.
Tool Calling & AgentsBuilding Fail-Safe Tool-Enabled Agents: Pydantic Schema Enforcement, Sandboxed Execution, and Error Recovery
LLM tool calling is fundamentally probabilistic. When an agent emits malformed arguments or calls destructive APIs in a blind loop, systems fail. Here is our blueprint for deterministic schema validation, self-correcting retries, and isolation.