Engineering AI Evaluation Frameworks: Continuous Benchmarking with LLM-as-a-Judge and Ragas in CI/CD

The Death of "Vibes-Based" AI Development

In early 2023, most AI engineers evaluated systems through "vibes": tweaking a system prompt, running 3 questions in a playground, deciding it "looked better," and deploying to production.

Two days later, users reported hallucinations in edge cases that were never tested.

In mature production AI systems, every prompt adjustment, chunking change, embedding swap, or model migration must be validated against a curated Golden Dataset with automated scorecards.

The Core Metric Triad: Faithfulness, Relevance, Recall

When evaluating RAG and agent systems, break down the pipeline into isolated components:

       [ User Query ]
             │
             ▼
     [ Retrieval Phase ] ───► Evaluated by: Context Recall & Context Precision
             │
             ▼
    [ Generation Phase ] ───► Evaluated by: Faithfulness & Answer Relevance

1. Faithfulness (Hallucination Detector)

Does the generated response stem exclusively from the retrieved context? If the context says nothing about a date, but the model answers with a date, Faithfulness drops to 0.

2. Answer Relevance

Did the model actually answer what was asked, or did it digress into unrelated verbose prose?

3. Context Recall

Did the retrieval engine successfully fetch the ground-truth facts required to answer the question?

Building Automated Eval Pipelines with Ragas

Here is an automated evaluation suite using Python and Ragas:

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevance,
    context_recall,
    context_precision,
)

eval_dataset = Dataset.from_dict({
    "question": [
        "What is the maximum context length of model X?",
        "How do I configure OAuth in the backend?"
    ],
    "contexts": [
        ["Model X supports up to 128,000 tokens in standard context window."],
        ["Configure OAuth by setting CLIENT_ID and CLIENT_SECRET in .env."]
    ],
    "answer": [
        "Model X has a context window of 128k tokens.",
        "Set CLIENT_ID and CLIENT_SECRET in your .env configuration."
    ],
    "ground_truth": [
        "128,000 tokens",
        "Configure CLIENT_ID and CLIENT_SECRET in the .env environment file."
    ]
})

results = evaluate(
    eval_dataset,
    metrics=[
        faithfulness,
        answer_relevance,
        context_recall,
        context_precision
    ]
)

print(results)
# {'faithfulness': 0.98, 'answer_relevance': 0.95, 'context_recall': 0.92, ...}

GitHub Actions PR Gating for Prompts

We treat system prompts and retrieval parameters exactly like backend source code: 1. Every PR touching prompts/ or retrieval/ runs a subset of 100 Golden benchmark questions. 2. If average faithfulness < 0.92 or context_recall degrades by more than 2%, CI fails and prevents merging. 3. The evaluation summary posts a visual Markdown diff table directly onto the GitHub pull request.