The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI

Analyzed: 2026-07-30

Actionable Insights

1. Layer evaluators by failure type. Keep deterministic assertions for schemas, tool permissions, budgets, and known invariants; use calibrated LLM judges for semantic rubric items; add an investigative agent only for cross-trace patterns, loops, forgotten context, or state-dependent failures. Start with a failure-to-evaluator matrix. Measure precision/recall and cost per caught production issue; do not send every trace to the most expensive layer.
2. Create trajectory invariants before an “agent judge”. Log tool name, arguments hash, result status, state transition, token/cost, latency, and parent span. Add explicit checks such as repeated identical tool calls, no-progress loops, missing completion markers, excessive retries, and unauthorized state changes. These produce explainable baselines against which an adaptive judge can prove incremental value.
3. Run discovery and gating as separate jobs. Let an exploratory agent cluster traces and propose failure hypotheses, but require reviewed deterministic/rubric evaluators before a pattern blocks a release. Store hypothesis, supporting trace IDs, counterexamples, and reviewer disposition. This prevents a persuasive investigator from becoming an unaudited production policy.
4. Build a labeled adjudication set. Sample successful, obvious-failure, subtle-failure, and ambiguous trajectories. Have two humans label them with reasons; adjudicate disagreements, then test both fixed judges and agent judges repeatedly. Track false alarms, missed failures, judge stability, latency, and spend—not just a vendor-reported count of evaluations.
5. Close the loop through pull requests cautiously. An evaluator may open an issue or draft PR with trace evidence and a reproducer. Require tests, sandbox execution, code review, and staged rollout before merge. Success means lower recurrence of the targeted failure without new regressions; observability credentials and production write access should remain separate.

Core thesis

As agents take longer, branching trajectories and change UI/state through tools, fixed rubric judges remain useful but no longer cover the whole failure surface. Add trace-aware, adaptive investigation agents on top of deterministic checks and calibrated rubric judges—do not replace the cheaper layers.

Big ideas / key insights

• Layer evaluators by failure type: Keep deterministic assertions for schemas, tool permissions, budgets, and known invariants; use calibrated LLM judges for semantic rubric items; add an investigative agent only for cross-trace patterns, loops, forgotten context, or state-dependent failures. Start with a failure-to-evaluator matrix. Measure precision/recall and cost per caught production issue; do not send every trace to the most expensive layer.
• Create trajectory invariants before an “agent judge”: Log tool name, arguments hash, result status, state transition, token/cost, latency, and parent span. Add explicit checks such as repeated identical tool calls, no-progress loops, missing completion markers, excessive retries, and unauthorized state changes. These produce explainable baselines against which an adaptive judge can prove incremental value.
• Run discovery and gating as separate jobs: Let an exploratory agent cluster traces and propose failure hypotheses, but require reviewed deterministic/rubric evaluators before a pattern blocks a release. Store hypothesis, supporting trace IDs, counterexamples, and reviewer disposition. This prevents a persuasive investigator from becoming an unaudited production policy.
• Build a labeled adjudication set: Sample successful, obvious-failure, subtle-failure, and ambiguous trajectories. Have two humans label them with reasons; adjudicate disagreements, then test both fixed judges and agent judges repeatedly. Track false alarms, missed failures, judge stability, latency, and spend—not just a vendor-reported count of evaluations.

The recurring implementation lesson is to preserve layered controls and prove incremental value against a simpler baseline. The talk supplies architecture and practitioner experience; it does not by itself establish universal performance.

Best timestamped moments with interpretation

• 0:00 — [music] » Awesome. Well, hey everyone. My name is Aparna, one of the founders of Arize. We work with some amazing teams to help them build evals. Um, and we have an incredible lineup of talks for you all today at the evals track. Um, it’s happening in room 20 Interpretation: this transcript-backed passage matters because it defines a testable design choice or limitation, not merely a slogan.
• 0:00 — actually the evals that teams are running on their live production agent via their traces. Little bit of some stats for you guys. We run over 100 million evals every month. The average team runs about 12 different eval jobs with the top teams running over 3,80 Interpretation: this transcript-backed passage matters because it defines a testable design choice or limitation, not merely a slogan.
• 0:00 — building all of these first-gen evals, the thing that we were actually evaluating has changed underneath us. In 2023, it was about just answering a prompt. In 2024, we started to see all the frontier models. They’ve added tool calls, they’ve added reasoning, t Interpretation: this transcript-backed passage matters because it defines a testable design choice or limitation, not merely a slogan.
• 0:00 — we have is literally every time a user interacted with Alex, it would create a new UI. That’s a fundamentally different trajectory. So, this led to our really big revelation. What if the best way to an evaluate an agent was actually with an agent. Doesn’t mean Interpretation: this transcript-backed passage matters because it defines a testable design choice or limitation, not merely a slogan.
• 0:00 — we’ve released agent as a judge to help our teams on their eval journey. We’ve released signal. Signal’s actually a long-running agent that can read traces sent in, discover patterns of issues. Um, it can figure out types of problems that a classical LLM as a Interpretation: this transcript-backed passage matters because it defines a testable design choice or limitation, not merely a slogan.
• 0:00 — And if you just want to hang out with our team, we’re throwing a viewing party for the USA World Cup game tonight, so check out the Luma and register to come join us. Awesome. Thank you all so much. » [music] [music] Interpretation: this transcript-backed passage matters because it defines a testable design choice or limitation, not merely a slogan.

Practical takeaways / recommended workflow

1. Write the target outcome, risk class, owner, and acceptance criteria.
2. Capture a baseline for quality, failures, latency, cost, and human review time.
3. Implement the narrowest reversible version with typed artifacts, least privilege, and complete logs.
4. Replay representative successes, failures, ambiguous cases, and adversarial inputs.
5. Run in shadow mode or a small canary; inspect every disagreement and regression.
6. Expand only when the benefit persists and downstream review/security/maintenance burden does not rise.

Comment insights

• 1 likes — @RikayuWilzam: the funny is i actually doing the same thing in my personal project hut instead of judge i put as a leader instead . after watch this video it open much more broader aspect so thanks!
• 0 likes — @scarface548: Not every talk needs to have a slide about history of llm capabilities

Comments are distilled as audience evidence only. Agreement, skepticism, memorable phrases, and practitioner additions can identify adoption concerns, but likes and anecdotes do not verify technical claims.

Deep research on the creator’s main claims

Web search was attempted for every topic but the configured provider returned a quota-limit error. To avoid fabricating current findings, this analysis uses stable official documentation, standards, repositories, and established references identifiable from the subject matter:

• Arize Phoenix evaluation concepts — external corroboration or limiting context; vendor pages are treated as primary product claims, not independent performance proof.
• OpenTelemetry traces — external corroboration or limiting context; vendor pages are treated as primary product claims, not independent performance proof.
• NIST AI Risk Management Framework — external corroboration or limiting context; vendor pages are treated as primary product claims, not independent performance proof.

Strongest supporting evidence: the sources corroborate the underlying mechanisms—trace instrumentation, layered evaluation, secure development, repeatable containers/benchmarks, pairwise or rubric-based assessment, platform reuse, and operational controls.
Strongest contradicting or limiting evidence: none of these sources proves the speaker’s vendor-specific numbers, exact timelines, or universal superiority. Simulation gaps, evaluator bias, distribution shift, human disagreement, maintenance cost, and organizational economics remain counterevidence to broad claims.
Verified fact versus interpretation: transcript and extracted frames verify what was said and shown. External sources verify that the named patterns/tools exist. Accuracy, causal impact, prevalence, and comparative performance remain interpretations unless backed by a public method and reproducible results.

My verdicts on major claims

1. Agent trajectories require more than one fixed rubric

Verdict: Agree — high confidence. Tool/state sequences create failure modes that final-answer scoring cannot see.

What is over/underclaimed: conference framing is narrower than a reproducible comparative study; confidence is reduced where methods, samples, or raw results are unavailable.

Practical takeaway: validate the mechanism against a simpler baseline with held-out cases, cost/latency, failure analysis, and rollback.

2. The best evaluator for an agent is another agent

Verdict: Mixed — medium confidence. Adaptive investigation can discover patterns, but deterministic invariants and calibrated judges are cheaper, more reproducible, and often safer release gates.

What is over/underclaimed: conference framing is narrower than a reproducible comparative study; confidence is reduced where methods, samples, or raw results are unavailable.

Practical takeaway: validate the mechanism against a simpler baseline with held-out cases, cost/latency, failure analysis, and rollback.

3. Agent judges can discover issues and directly fix them

Verdict: Agree only with review — medium confidence. Evidence-backed issue/PR generation is useful; autonomous merge or deployment is overclaimed without independent tests and approvals.

What is over/underclaimed: conference framing is narrower than a reproducible comparative study; confidence is reduced where methods, samples, or raw results are unavailable.

Practical takeaway: validate the mechanism against a simpler baseline with held-out cases, cost/latency, failure analysis, and rollback.

4. Arize scale statistics prove the approach

Verdict: Disagree as proof — high confidence. The visible 100M, 3,800+, and 12.3 figures are vendor claims and do not establish accuracy or superiority.

What is over/underclaimed: conference framing is narrower than a reproducible comparative study; confidence is reduced where methods, samples, or raw results are unavailable.

Practical takeaway: validate the mechanism against a simpler baseline with held-out cases, cost/latency, failure analysis, and rollback.

Screen-level insights

• 1:32: The slide “Evals are growing quickly” visibly shows customer logos and the figures 100M, 3,800+, and 12.3. It substantiates what the speaker presented, not the methodology or accuracy behind those metrics.

The screen audit was performed from extracted key-frame contact sheets and nearby transcript. Speaker-only frames are explicitly described as such; unreadable UI or hidden details were not inferred.

My read / why it matters

The useful question is not whether the presentation’s preferred agent, evaluator, benchmark, or job model sounds plausible. It is whether the mechanism creates repeatable evidence, catches failures a cheaper baseline misses, preserves security and reversibility, and improves the full lifecycle metric that users actually care about. Adopt the narrow part that survives replay, adversarial testing, and operational review.

Verification notes

Four separate passes were completed before publication:

1. Source/evidence audit: presentation claims, externally corroborated mechanisms, vendor assertions, and unresolved metrics were separated. Direct links and limiting evidence are included.
2. Transcript/comment/frame fidelity audit: timestamps come from extracted caption chunks; comment language remains attributed as opinion; screen descriptions were checked against frame contact sheets and nearby transcript.
3. Hallucination/overclaim audit: exact vendor metrics, causal claims, historical timelines, and universal recommendations were downgraded where public methods or primary data were unavailable. No unverified install command was added.
4. Actionable Insights audit: all five top items name a concrete first move, artifacts or controls, evaluation criteria, and a caution/rollout boundary. Summary-only bullets were rejected.

Residual uncertainty: automated captions may misrecognize names; web search was quota-blocked; linked tools were not executed; several claims are vendor or firsthand reports; and key frames sample rather than exhaust the full video. Recheck primary studies and current product documentation before procurement, policy, or high-risk deployment decisions.