Judging Agents: LLM Judges, Jev, and System One Models

Most of the judgements inside an agent eval are small, bounded decisions. Here is how code, LLM judges, and System One models such as Jev each fit, and what the evidence says about them.

By Vaibhav Khamesra · October 31, 2026 · 9 min read

Why agents need a different kind of grader

A single LLM call has an input and an output. An agent has a trajectory: it reads, plans, calls tools, changes state, and talks to a user over many turns. Anthropic's engineering guide to agent evals separates the transcript (everything the agent said and did) from the outcome (the final state of the environment). A flight-booking agent can say "your flight is booked" while no reservation exists in the database. Good evals grade both.

The same guide describes three grader families:

The rest of this article is about the middle row, and about a newer option that sits between code and LLMs.

LLM-as-a-judge: what the evidence says

The foundational study is Zheng and colleagues' 2023 paper introducing MT-Bench and Chatbot Arena. They found that strong judges such as GPT-4 matched both controlled and crowdsourced human preferences with over 80% agreement, the same level at which humans agree with each other. They also documented the biases that still matter today:

• Position bias: preferring whichever answer appears first (or second) in a pairwise comparison.
• Verbosity bias: preferring longer answers regardless of quality.
• Self-enhancement bias: preferring answers written by the same model family.
• Limited reasoning: grading maths or logic it cannot itself solve.

Practical mitigations that have held up in production:

• Swap the order in pairwise comparisons and only count consistent verdicts.
• Grade one dimension per judge call with a clear rubric. Anthropic recommends isolated judges per dimension and giving the judge a way out, such as returning "Unknown" when information is missing.
• Prefer binary pass/fail labels to 1-to-10 scales. They are easier for humans to label consistently and easier to calibrate.
• Give the judge the evidence: retrieved documents, tool results, and the policy the agent was meant to follow.

Decompose before you judge

Asking a model "score this answer from 0 to 1" produces a number with no justification. A better pattern is to break the judgement into checkable parts and compute the score in code. DeepEval's Faithfulness metric is a clear example: an LLM extracts the claims in the answer and the facts in the retrieved context, each claim is judged as supported or not, and the score is computed:

The extraction step is genuinely a language task. The verdict step ("is this claim supported?") is a bounded decision that only needs a yes, a no, or "borderline". That distinction is what System One models are built around.

Agents judging agents

For long, tool-heavy tasks, a single judge call may not see enough. Zhuge and colleagues' Agent-as-a-Judge framework (2024) gives the judge its own tools to inspect the workspace and intermediate steps. On DevAI, a benchmark of 55 AI-development tasks with 365 hierarchical requirements, they report that Agent-as-a-Judge dramatically outperformed LLM-as-a-Judge and was as reliable as their human evaluation baseline. The trade-off is cost: an agentic judge runs many calls per evaluation.

System One models: typed decisions instead of generated text

TypeSafe AI describes Jev as the first "System One" model, a name borrowed from Daniel Kahneman's fast, intuitive System 1 thinking. Like an LLM, it reads natural-language input. Unlike an LLM, it does not generate text. You give it a state (the content to judge) and a set of bounded questions, and it returns typed answers with probabilities. There are three question primitives:

For a Score with levels ordered worst first, the result is the expected position across levels, which lets an answer land between "mostly grounded" and "fully grounded" rather than snapping to one:

Because the option set comes from your code, the model cannot invent a label, change its casing, or bury the verdict in an explanation. There is no JSON to repair. Questions sent together are evaluated in parallel against the same state.

What a System One judge looks like

Here is a sketch of judging one support-agent trace with the TypeSafe Python SDK. The questions are bounded and phrased so that a high value means yes, as TypeSafe's documentation recommends; thresholds live in your code.

from typesafe_sdk import Noul, NoulCriteria, TypeSafeClient

QUESTIONS = {
    "grounded_in_tools": Noul(
        instructions=(
            "Is every factual statement in the final reply "
            "supported by the tool results in the trace?"
        ),
    ),
    "followed_refund_policy": Noul(
        instructions="Did the agent follow the refund policy included in the state?",
    ),
    "escalated_when_asked": Noul(
        instructions=(
            "If the customer asked for a human, "
            "did the agent hand over to a human?"
        ),
        criteria=NoulCriteria(
            true=(
                "The customer asked for a person and the agent "
                "escalated, or the customer never asked"
            ),
            false=(
                "The customer asked for a person and the agent "
                "kept the conversation"
            ),
        ),
    ),
}

YES, NO = 0.7, 0.3

def judge(trace: dict, policy: str) -> dict:
    with TypeSafeClient() as client:
        response = client.system_one(
            model="jev-latest",
            state={"trace": trace, "refund_policy": policy},
            questions=QUESTIONS,
        )
    verdicts = {}
    for name, answer in response.answers.items():
        p = answer.noul
        verdicts[name] = "pass" if p >= YES else "fail" if p <= NO else "review"
    return verdicts

Anything between the thresholds goes to a person. TypeSafe suggests 0.7 and 0.3 as starting points and warns against fitting thresholds closely to a handful of examples; set them from a labelled sample of your own traces.

The evidence so far

System One models are new, so read the evidence carefully. Here is what has been published.

Vendor integration (DeepEval). DeepEval added Jev as an opt-in mode. In hybrid mode the LLM still extracts claims and Jev decides the verdicts; in system_one mode Jev judges the whole test case with no LLM call. DeepEval cites Jev's price at $0.042 per million input tokens with no output-token charge, and most queries completing in roughly 100 ms. It also notes that a system_one score is a different judge of the same metric, not an interchangeable number.

LangChain's experiment. LangChain compared Jev with GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 as judges of a weather agent, using five captured runs, 100 repetitions per judge, and a human reviewer's labels as the oracle. On the binary pass/fail decision, Jev matched the oracle on all 500 repeated decisions, against 99.8% for Terra, 96.4% for Luna, and 80.0% for Claude. Jev's quality-score variance was 92 to 913 times lower than the LLM judges, at an average of 0.44 seconds and $0.00035 per call. The authors themselves call the test narrow and early, and the LLM judges ran at provider default settings.

Independent benchmarks. A paired study by Jiawei Li evaluated Jev and the open-weight System One model Laya on 11 agent decision points built from 18 public sources, with 7,283 base cases and 6,640 robustness variants. Jev was significantly more accurate on 9 of the 11 decision points, but neither model beat chance on zero-shot model routing. The same paper audited its own pipeline and found errors that had inflated headline claims, including gate accuracy reported as end-to-end quality and thresholds tuned in-sample.

A second study benchmarked Jev 1.13.0 zero-shot on 37 public datasets, 346,009 requests in total for under US$10. Jev scored higher than Qwen3.8-27B on 27 datasets, lower on nine, and tied on one.

System1-Arena, a benchmark of 30 models across 105 tasks, found that the largest System One models match or exceed fast closed LLMs on non-agentic decisions, with better-calibrated probabilities at a quarter of the latency. A reasoning LLM was most accurate, at about 70 times the cost of a fast LLM. In closed-loop environments where a model must act repeatedly, small System One models rarely succeeded.

The pattern across these results: System One models are strong judges of bounded decisions and weak agents. Use them to grade, gate, and route, and keep open-ended reasoning with LLMs.

A layered judging stack

Putting this together, a practical evaluator for an agent trace looks like this:

1. Code checks first. Did the required tool calls happen with valid parameters? Is the final database state correct? Did the reply leak an internal ID? These are free and deterministic.
2. Bounded verdicts with a System One model. Groundedness per claim, policy compliance, escalation correctness, intent labels. Each returns a probability you can threshold and track over time.
3. LLM judges for open-ended quality. Tone, helpfulness of the explanation, whether a research summary is complete. Use a rubric per dimension and keep the critique text for debugging.
4. Humans to calibrate. Label a sample of traces every release. Compare every automated judge against those labels and watch for drift.

Calibrating any judge

A judge is itself a model, so it needs its own small evaluation. Hamel Husain's advice from production work is worth repeating: raw agreement is misleading when failures are rare. If 95% of traces pass, a judge that always says "pass" agrees 95% of the time and catches nothing. Measure instead:

• True positive rate: of the traces humans marked as failures, how many did the judge flag?
• True negative rate: of the traces humans marked as good, how many did the judge pass?
• Repeatability: run the judge several times on the same trace. How often does the verdict change?

Re-run this calibration when you change the judge's model, prompt, or thresholds, and when the agent's behaviour changes significantly. Cheap judges amplify mistakes as easily as they amplify signal: a consistently wrong evaluator produces bad feedback at scale.

Where to start

• Pull 20 to 50 real traces, including known failures, and label them pass or fail by hand.
• Write code checks for everything objectively checkable.
• Turn the remaining judgements into bounded questions. Try a System One model and an LLM judge on the same questions and compare both with your labels.
• Keep the judge with the best true positive and true negative rates for each question, and record why.