The Testing Paradox of Non-Deterministic Code
In traditional backend engineering, testing is straightforward:
def test_calculate_tax():
assert calculate_tax(100.0) == 15.0If you apply this paradigm to an LLM:
def test_summarize_policy():
# FAILS: The wording changes on every invocation even with temperature=0
assert summarize_policy(doc) == "The return window is 30 days."Because foundation models sample from token probability distributions, exact match assertions trigger constant false alarms, causing teams to abandon testing altogether.
The Three-Tier Assertion Hierarchy
To build resilient regression suites, we separate assertions into three distinct levels:
1. Deterministic Structural Assertions (Fast, 0 Cost)
├── Is valid JSON?
├── Does it conform to Pydantic schema?
└── Are required keys present?
2. Semantic Embedding Assertions (Fast, Low Cost)
├── Cosine similarity to Golden Answer >= 0.88
└── Entity overlap (Jaccard similarity of extracted entities)
3. LLM-as-a-Judge Rubric Assertions (Deep, Medium Cost)
├── Does the answer satisfy all 4 criteria in the grading rubric?
└── Does it contain any hallucinated assertions?Building Semantic Pytest Fixtures
Here is our production Pytest test structure:
import pytest
from sentence_transformers import SentenceTransformer
import numpy as np
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")
def compute_cosine_similarity(text_a: str, text_b: str) -> float:
emb_a = embedding_model.encode(text_a)
emb_b = embedding_model.encode(text_b)
return float(np.dot(emb_a, emb_b) / (np.linalg.norm(emb_a) * np.linalg.norm(emb_b)))
@pytest.mark.asyncio
async def test_agent_refund_policy_reasoning():
user_query = "Can I return a opened laptop after 25 days?"
# Execute agent
response = await run_support_agent(user_query)
# Tier 1: Structural check
assert response.status == "success"
assert response.resolution in ["approved", "rejected", "escalate"]
# Tier 2: Semantic check against Golden reference
golden_answer = "Yes, items can be returned within 30 days, but open boxes incur a 15% restocking fee."
similarity = compute_cosine_similarity(response.message, golden_answer)
assert similarity >= 0.85, f"Semantic drift detected. Score: {similarity}"
# Tier 3: Critical constraint check (Regex/Deterministic)
assert "restocking fee" in response.message.lower()Monitoring Model Checkpoint Drift
When LLM providers quietly update underlying model weights (e.g., pointing gpt-4o or claude-3-5-sonnet to a new internal checkpoint), prompt outputs can drift without any code changes on your end.
Running this suite nightly on scheduled cron builds detects behavioral regressions before customers experience degraded service.
Enjoyed this engineering playbook?
Explore our other in-depth articles on production AI.
Related Articles
View All →
Multi-Agent SystemsArchitecting Multi-Agent Systems in Production: Supervisor vs Peer-to-Peer Orchestration with LangGraph
Single LLM agents struggle with context window pollution and hallucination cascades when tasks exceed 5-10 reasoning steps. Here is how we architect production multi-agent systems using supervisor-worker patterns, typed shared state, and deterministic cyclic graphs.
Advanced RAGBeyond Naive RAG: Building Production Hybrid Retrieval with Dense Vectors, BM25, and Cross-Encoder Rerankers
Pure vector search fails when queries contain exact product SKUs, code identifiers, or acronyms. We dive deep into Reciprocal Rank Fusion (RRF), Cross-Encoder reranking, and chunking heuristics that achieve 96%+ context precision.
Tool Calling & AgentsBuilding Fail-Safe Tool-Enabled Agents: Pydantic Schema Enforcement, Sandboxed Execution, and Error Recovery
LLM tool calling is fundamentally probabilistic. When an agent emits malformed arguments or calls destructive APIs in a blind loop, systems fail. Here is our blueprint for deterministic schema validation, self-correcting retries, and isolation.