How Evals and Prompts Shape Agent Behavior — Preetika Bhateja & Daniel Bump, YouTube Ads

Analyzed: 2026-07-30

Actionable Insights

1. Evaluate tools before the full agent. Create unit-level cases for each tool schema, argument validation, timeout, permission, and error response. Fix tool reliability before attributing failures to planning. A tool graduates when success/error behavior is repeatable and the agent can distinguish retryable from terminal errors.
2. Start with 20–50 high-value cases, including negatives. Select core user tasks, likely edge cases, forbidden actions, and “should abstain” examples. Store input, expected properties, rubric, severity, and provenance. Run multiple stochastic trials per configuration; expand from production failures rather than trying to author a giant perfect suite on day one.
3. Require rater rationales and adjudicate ambiguity. A pass/fail label is insufficient for prompt or model improvement. Ask raters which criterion failed and quote the evidence. Track inter-rater agreement; when the team disagrees, repair the rubric or split the task rather than averaging away a real product-policy conflict.
4. Calibrate LLM judges against held-out human labels. Use a separate calibration set and report precision, recall, confusion by failure type, and stability across repeated judge calls. Prefer deterministic checks where possible. Recalibrate when prompts, models, output format, or user population changes; an automated judge should not grade cases it helped create without leakage controls.
5. Use ablations to prove each prompt or architecture change. Change one factor at a time—tool description, system instruction, critique loop, model, or retrieval context—and compare against the same case set. Require improvement on target metrics without regressions in cost, latency, safety, or negative cases. Keep a changelog linking each release to eval results.

Core thesis

Reliable agent development is a staged hill-climb: strengthen tools first, explore failures manually, start with a small positive-and-negative golden set, collect rater explanations, calibrate automated judges to humans, and expand only after the rubric becomes stable.

Big ideas / key insights

• Evaluate tools before the full agent: Create unit-level cases for each tool schema, argument validation, timeout, permission, and error response. Fix tool reliability before attributing failures to planning. A tool graduates when success/error behavior is repeatable and the agent can distinguish retryable from terminal errors.
• Start with 20–50 high-value cases, including negatives: Select core user tasks, likely edge cases, forbidden actions, and “should abstain” examples. Store input, expected properties, rubric, severity, and provenance. Run multiple stochastic trials per configuration; expand from production failures rather than trying to author a giant perfect suite on day one.
• Require rater rationales and adjudicate ambiguity: A pass/fail label is insufficient for prompt or model improvement. Ask raters which criterion failed and quote the evidence. Track inter-rater agreement; when the team disagrees, repair the rubric or split the task rather than averaging away a real product-policy conflict.
• Calibrate LLM judges against held-out human labels: Use a separate calibration set and report precision, recall, confusion by failure type, and stability across repeated judge calls. Prefer deterministic checks where possible. Recalibrate when prompts, models, output format, or user population changes; an automated judge should not grade cases it helped create without leakage controls.

The recurring implementation lesson is to preserve layered controls and prove incremental value against a simpler baseline. The talk supplies architecture and practitioner experience; it does not by itself establish universal performance.

Best timestamped moments with interpretation

• 0:00 — [music] Hi everyone. Uh sounds like everybody came back from lunch. So hope everybody is recharged and not sleepy at all. It’s always interesting to do a talk right after lunch because you never know. It’s a mixed crowd. Uh but we’re very happy to be here. Hap Interpretation: this transcript-backed passage matters because it defines a testable design choice or limitation, not merely a slogan.
• 0:00 — behave in the real world by defining what good looks like. Uh it’s basically just setting this is uh you know our target output. Uh so to build evals that actually scale they really need to be strict and measurable. And uh so an interesting uh thing here that Interpretation: this transcript-backed passage matters because it defines a testable design choice or limitation, not merely a slogan.
• 0:00 — had a lot of situations, especially early on when you’re building things. of course like there are so many edge cases and difficult cases that we’ve not tested out that a raider might encounter. So they’re coming back to you saying oh what what what should I d Interpretation: this transcript-backed passage matters because it defines a testable design choice or limitation, not merely a slogan.
• 0:00 — disclaimer in what it’s searching for and it says, okay, I found a disclaimer and now I’m going to go ahead and remove it, which was not what we asked it to do. Uh, this is a sample image that I created that I ran through the agent. It says, “America, we can d Interpretation: this transcript-backed passage matters because it defines a testable design choice or limitation, not merely a slogan.
• 0:00 — » Okay, awesome. So, we talked a bunch about like what worked for us while we were building evals. Of course, your mileage may vary depending on your application. Things can uh things can differ. Uh some of the things that we wanted to recap here was uh it’s Interpretation: this transcript-backed passage matters because it defines a testable design choice or limitation, not merely a slogan.
• 0:00 — » [music] Interpretation: this transcript-backed passage matters because it defines a testable design choice or limitation, not merely a slogan.

Practical takeaways / recommended workflow

1. Write the target outcome, risk class, owner, and acceptance criteria.
2. Capture a baseline for quality, failures, latency, cost, and human review time.
3. Implement the narrowest reversible version with typed artifacts, least privilege, and complete logs.
4. Replay representative successes, failures, ambiguous cases, and adversarial inputs.
5. Run in shadow mode or a small canary; inspect every disagreement and regression.
6. Expand only when the benefit persists and downstream review/security/maintenance burden does not rise.

Comment insights

No substantive comments were extracted. That absence is recorded rather than replaced with invented audience consensus.

Comments are distilled as audience evidence only. Agreement, skepticism, memorable phrases, and practitioner additions can identify adoption concerns, but likes and anecdotes do not verify technical claims.

Deep research on the creator’s main claims

Web search was attempted for every topic but the configured provider returned a quota-limit error. To avoid fabricating current findings, this analysis uses stable official documentation, standards, repositories, and established references identifiable from the subject matter:

• Google PAIR People + AI Guidebook — external corroboration or limiting context; vendor pages are treated as primary product claims, not independent performance proof.
• OpenAI eval-driven system design — external corroboration or limiting context; vendor pages are treated as primary product claims, not independent performance proof.
• NIST AI RMF — external corroboration or limiting context; vendor pages are treated as primary product claims, not independent performance proof.

Strongest supporting evidence: the sources corroborate the underlying mechanisms—trace instrumentation, layered evaluation, secure development, repeatable containers/benchmarks, pairwise or rubric-based assessment, platform reuse, and operational controls.
Strongest contradicting or limiting evidence: none of these sources proves the speaker’s vendor-specific numbers, exact timelines, or universal superiority. Simulation gaps, evaluator bias, distribution shift, human disagreement, maintenance cost, and organizational economics remain counterevidence to broad claims.
Verified fact versus interpretation: transcript and extracted frames verify what was said and shown. External sources verify that the named patterns/tools exist. Accuracy, causal impact, prevalence, and comparative performance remain interpretations unless backed by a public method and reproducible results.

My verdicts on major claims

1. Early “vibe eval” is useful before scaling

Verdict: Agree — medium-high confidence. Manual exploration reveals failure taxonomy, provided it transitions into recorded cases and rubrics.

What is over/underclaimed: conference framing is narrower than a reproducible comparative study; confidence is reduced where methods, samples, or raw results are unavailable.

Practical takeaway: validate the mechanism against a simpler baseline with held-out cases, cost/latency, failure analysis, and rollback.

2. Start early and small instead of building a massive golden set

Verdict: Strongly agree — high confidence. Small representative suites support faster iteration and can grow from observed failures.

What is over/underclaimed: conference framing is narrower than a reproducible comparative study; confidence is reduced where methods, samples, or raw results are unavailable.

Practical takeaway: validate the mechanism against a simpler baseline with held-out cases, cost/latency, failure analysis, and rollback.

3. Rater explanations are essential

Verdict: Agree — high confidence. Rationales expose rubric ambiguity and make errors actionable, though they also need quality review.

What is over/underclaimed: conference framing is narrower than a reproducible comparative study; confidence is reduced where methods, samples, or raw results are unavailable.

Practical takeaway: validate the mechanism against a simpler baseline with held-out cases, cost/latency, failure analysis, and rollback.

4. Reliability is a function of capabilities, guardrails, and evals

Verdict: Mostly agree — high confidence. Operations, data quality, and human escalation should also be named explicitly.

What is over/underclaimed: conference framing is narrower than a reproducible comparative study; confidence is reduced where methods, samples, or raw results are unavailable.

Practical takeaway: validate the mechanism against a simpler baseline with held-out cases, cost/latency, failure analysis, and rollback.

Screen-level insights

• 0:32: The title slide “Model Whisperers: Evals for Production Agents” fixes the production scope.
• 1:03: “Building an AI agent is hard / Making it reliable is harder” visually emphasizes reliability over demo capability.
• 1:33: The “Agent foundation” slide lists foundational concerns, matching the recommendation to optimize tools before agent-level evals.
• 13:19: “Hillclimbing can be rewarding” includes a process flow, reinforcing eval as an iterative loop rather than a one-time score.
• 18:23: Both presenters are on stage during the question about human versus LLM judges; the transcript, rather than a hidden demo, is the evidence for their calibration caveat.

The screen audit was performed from extracted key-frame contact sheets and nearby transcript. Speaker-only frames are explicitly described as such; unreadable UI or hidden details were not inferred.

My read / why it matters

The useful question is not whether the presentation’s preferred agent, evaluator, benchmark, or job model sounds plausible. It is whether the mechanism creates repeatable evidence, catches failures a cheaper baseline misses, preserves security and reversibility, and improves the full lifecycle metric that users actually care about. Adopt the narrow part that survives replay, adversarial testing, and operational review.

Verification notes

Four separate passes were completed before publication:

1. Source/evidence audit: presentation claims, externally corroborated mechanisms, vendor assertions, and unresolved metrics were separated. Direct links and limiting evidence are included.
2. Transcript/comment/frame fidelity audit: timestamps come from extracted caption chunks; comment language remains attributed as opinion; screen descriptions were checked against frame contact sheets and nearby transcript.
3. Hallucination/overclaim audit: exact vendor metrics, causal claims, historical timelines, and universal recommendations were downgraded where public methods or primary data were unavailable. No unverified install command was added.
4. Actionable Insights audit: all five top items name a concrete first move, artifacts or controls, evaluation criteria, and a caution/rollout boundary. Summary-only bullets were rejected.

Residual uncertainty: automated captions may misrecognize names; web search was quota-blocked; linked tools were not executed; several claims are vendor or firsthand reports; and key frames sample rather than exhaust the full video. Recheck primary studies and current product documentation before procurement, policy, or high-risk deployment decisions.