From Unit Tests to Whole-System Evals: Testing AI Agents

A testing pyramid for agents: fast deterministic tests on every commit, component evals on every pull request, and whole-system runs that measure reliability, not luck.

By Vaibhav Khamesra · October 31, 2026 · 7 min read

The testing pyramid, adapted for agents

Hamel Husain's widely cited guide to LLM product evals describes three levels, ordered by cost: unit tests (assertions), human and model evaluation, and A/B testing. Run the first on every code change, the second on a set cadence, and the third only after significant product changes. That structure maps cleanly onto agents, with one addition in the middle: running the whole agent in an environment.

Level 1: unit tests

Test the software around the model

An agent is mostly code: tool definitions, argument validation, parsers, retry logic, state machines, and guardrails. These parts are deterministic and should be tested the same way as any other code, with the model replaced by a stub or a recorded response.

import pytest
from agent.tools import refund_tool
from agent.guards import redact_internal_ids
from agent.state import Conversation, Status

def test_refund_rejects_amount_over_policy_limit():
    with pytest.raises(ValueError):
        refund_tool.validate({"order_id": "A123", "amount": 10_000})

def test_internal_ids_never_reach_the_customer():
    reply = "Your order a1b2c3d4-0000-4000-8000-1234567890ab has shipped."
    assert "a1b2c3d4" not in redact_internal_ids(reply)

def test_handoff_to_human_closes_bot_session():
    convo = Conversation(status=Status.BOT)
    convo.hand_off(reason="customer_requested_human")
    assert convo.status is Status.HUMAN
    assert convo.bot_can_reply() is False

The internal-ID check is adapted from a real example in Husain's guide, where a regex asserted that UUIDs from CRM results never appeared in replies. Tests like these are fast, free, and catch a large share of production incidents.

Scoped tests with the model in the loop

Some unit tests need the real model: does the listing finder produce a query that returns exactly one result when one listing matches? Husain's approach is to break the product into features and scenarios, write a few inputs per scenario, and assert on the result. Two differences from conventional unit tests matter:

• You do not need a 100% pass rate. The acceptable rate is a product decision based on which failures you can tolerate.
• Track pass rates over time, outside the CI log, so you can see whether a prompt change helped or hurt.

Level 2: component evals

Between unit tests and full runs sit components that can be evaluated with labelled data:

• Retrieval: for each query, is the needed document in the top k results? Measure recall at k before blaming the generator.
• Routing and intent: a labelled set and a confusion matrix.
• Tool selection and arguments: given a state, does the agent pick the right tool with valid arguments? Test both directions: cases where it should call the tool and cases where it should not.
• Bounded judgements: groundedness or policy compliance on a single step, graded by a calibrated judge. See the article on judging agents.

Level 3: whole-system evals

Run the agent in an environment

A whole-system eval gives the agent a task, its real tools, and an isolated environment, then lets it run to completion. Anthropic's guide separates the agent harness (the system that lets a model act) from the evaluation harness (the infrastructure that runs tasks, records every step, grades, and aggregates). Its practical requirements:

• Isolation. Each trial starts from a clean environment. Shared state between trials causes correlated failures and can inflate scores; Anthropic observed a model gaining an unfair advantage by reading the git history of previous trials.
• Reference solutions. For each task, keep a known working output that passes all graders. It proves the task is solvable and the graders are configured correctly.
• Unambiguous tasks. Two domain experts should independently reach the same pass/fail verdict. With frontier models, a 0% pass rate across many trials usually means the task or grader is broken.

Simulate the user

Conversational agents need someone to talk to. τ-bench, from Yao and colleagues at Sierra, pairs the agent with a language model that plays a user following a hidden goal, in domains such as retail and airline support with real policies and APIs. Grading compares the database state at the end of the conversation with the annotated goal state, so the agent is judged on what it did, not what it said.

Grade the outcome, not the path

It is tempting to assert an exact sequence of tool calls. Anthropic found that approach too rigid: agents regularly find valid paths the eval designer did not anticipate. Grade the final state, require specific steps only where a policy demands them, and build in partial credit for multi-part tasks. A support agent that verifies the customer and identifies the problem but fails to issue the refund is better than one that fails immediately, and the scores should show it.

Measure reliability with pass^k

Agents are non-deterministic, so a single passing run proves little. Two metrics capture different needs:

• pass@k: the probability that at least one of k attempts succeeds. Useful when a user or system can pick the best of several attempts.
• pass^k: the probability that all k attempts succeed. This is the one that matters for customer-facing agents, where every conversation has to work.

If an agent succeeds on a task 75% of the time and trials are independent, pass^3 is 0.75³, about 42%. In the original τ-bench results, even gpt-4o succeeded on fewer than 50% of tasks, and its pass^8 in the retail domain was below 25%.

The unbiased estimator is simple to compute from n trials with c successes per task:

from math import comb

def pass_hat_k(n: int, c: int, k: int) -> float:
    """Probability that k trials drawn from n (with c successes) all succeed."""
    return comb(c, k) / comb(n, k) if n >= k else float("nan")

def suite_pass_hat_k(results: dict[str, tuple[int, int]], k: int) -> float:
    """results maps task_id to (trials, successes)."""
    return sum(pass_hat_k(n, c, k) for n, c in results.values()) / len(results)

Small suites are noisy. With 50 tasks, one task is two percentage points, so treat small movements with caution and look at which tasks changed rather than only the average.

Capability suites and regression suites

Anthropic distinguishes two kinds of suites. Capability evals ask what the agent can do and should start at a low pass rate, giving the team a hill to climb. Regression evals ask whether the agent still does what it used to and should stay near 100%. When a capability task is reliably solved, it graduates into the regression suite.

Watch for saturation. A suite at 100% detects regressions but gives no signal for improvement, and large capability gains show up as tiny score changes near the ceiling.

Read the transcripts

Graders have bugs. Anthropic reports that Claude Opus 4.5 initially scored 42% on CORE-Bench until a researcher found rigid grading that rejected "96.12" when expecting "96.124991…", ambiguous task specifications, and stochastic tasks that could not be reproduced. After fixes and a less constrained scaffold, the score was 95%. The only reliable defence is to read transcripts of failed trials and confirm that each failure is the agent's fault.

A CI layout that works

1. Every commit: unit tests with stubbed models and recorded fixtures. Seconds, free.
2. Every pull request: component evals and a small smoke set of whole-system tasks with one trial each.
3. Nightly: the full regression and capability suites with several trials per task; report pass^k, cost, latency, and tokens per task.
4. Before release: human review of a sample of transcripts and recalibration of model-based judges.
5. In production: online evals on sampled traces with cheap judges, plus user feedback, A/B tests for major changes, and new failures turned into test cases.

Start with 20 to 50 tasks drawn from real failures and the checks you already run by hand. Anthropic's guidance is explicit that this is enough to begin, because early changes have large effects. The suite grows from there, one production failure at a time.