From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI

Analyzed: 2026-07-30

Actionable Insights

1. Turn traces into versioned benchmark tasks. For each selected trace, write instruction.md, an environment (Dockerfile or docker-compose.yml), fixtures, metadata, and hidden verifier/oracle code. Remove secrets and user data. Pin service and dependency versions so model, prompt, tool, and harness variants face the same environment.
2. Prove task solvability with an oracle. Implement a known-good sequence or reference state and run it before admitting a task. The oracle should demonstrate required APIs, permissions, and data are present, not leak the answer to the agent. Reject flaky tasks and record expected cost/latency bounds.
3. Score full-system outcomes. Measure final state and artifacts, not only the final message. Add success, safety violations, tool errors, retries, latency, token/API cost, and recovery behavior. Keep evaluators stable while changing model, reasoning level, prompt, skills, or tools so comparisons remain apples-to-apples.
4. Continuously sample production without contaminating tests. Cluster new traces, prioritize novel/high-severity failures, redact them, and add reviewed tasks. Maintain train/development/held-out splits; prevent agents or prompt authors from seeing hidden verifier details. Track duplicate rate and benchmark freshness.
5. Use replay as a release gate, then shadow mode. Run benchmark suites on every material agent change; block statistically meaningful regressions. Follow with shadow traffic because simulation cannot reproduce every external dependency or user behavior. Promote only when offline and shadow metrics agree within declared tolerances.

Core thesis

Production traces reveal failures but do not provide controlled comparison. Convert representative traces into private, reproducible simulation tasks with a mini-production environment, hidden oracle/verifiers, and metrics for success, cost, latency, and retries; then use them as release gates and learning data.

Big ideas / key insights

• Turn traces into versioned benchmark tasks: For each selected trace, write instruction.md, an environment (Dockerfile or docker-compose.yml), fixtures, metadata, and hidden verifier/oracle code. Remove secrets and user data. Pin service and dependency versions so model, prompt, tool, and harness variants face the same environment.
• Prove task solvability with an oracle: Implement a known-good sequence or reference state and run it before admitting a task. The oracle should demonstrate required APIs, permissions, and data are present, not leak the answer to the agent. Reject flaky tasks and record expected cost/latency bounds.
• Score full-system outcomes: Measure final state and artifacts, not only the final message. Add success, safety violations, tool errors, retries, latency, token/API cost, and recovery behavior. Keep evaluators stable while changing model, reasoning level, prompt, skills, or tools so comparisons remain apples-to-apples.
• Continuously sample production without contaminating tests: Cluster new traces, prioritize novel/high-severity failures, redact them, and add reviewed tasks. Maintain train/development/held-out splits; prevent agents or prompt authors from seeing hidden verifier details. Track duplicate rate and benchmark freshness.

The recurring implementation lesson is to preserve layered controls and prove incremental value against a simpler baseline. The talk supplies architecture and practitioner experience; it does not by itself establish universal performance.

Best timestamped moments with interpretation

• 0:00 — [music] » Okay. Oh, yeah. Thanks everyone for coming. And I know this is the last session before lunch. So, thanks for staying here. Let’s make it smooth and with good vibes. Just as that said. And thanks that for introduction. And for inviting me. So, yeah. Interpretation: this transcript-backed passage matters because it defines a testable design choice or limitation, not merely a slogan.
• 0:00 — like new model. But when you release agent to production, you also care about more metrics. You care about cost for solving the task. You care about latency. You care about number of retries. And by running evaluation offline with in simulations, you can effec Interpretation: this transcript-backed passage matters because it defines a testable design choice or limitation, not merely a slogan.
• 0:00 — Yeah, I’m saying like, “Okay, simulation environment is just Dockerfile and a bunch of stuff.” But, it is useful because now you have repeatable way of running experiments in agent simulation. So, now let’s dive deeper into main parts of benchmark. First is en Interpretation: this transcript-backed passage matters because it defines a testable design choice or limitation, not merely a slogan.
• 0:00 — environment because it can understand that it’s in simulation and it can hack it. Task could be too simple and like our verifiers could be too broad and in this case agent will always pass even if it does something incorrectly. It could be that agent always fa Interpretation: this transcript-backed passage matters because it defines a testable design choice or limitation, not merely a slogan.
• 0:00 — And finally, uh, you want, uh, your, um, benchmark to be part of your, uh, agent ops, uh, loop. Thank you very much for for your time and, uh, I think maybe I will have answer for one question, but will be happy to answer any questions outside and please check Interpretation: this transcript-backed passage matters because it defines a testable design choice or limitation, not merely a slogan.
• 0:00 — marks it as like not solved correctly or like it marks that trace wasn’t optimal enough. And this is where you want to have subject matter expert that can basically tune the agent that does the review. » [music] Interpretation: this transcript-backed passage matters because it defines a testable design choice or limitation, not merely a slogan.

Practical takeaways / recommended workflow

1. Write the target outcome, risk class, owner, and acceptance criteria.
2. Capture a baseline for quality, failures, latency, cost, and human review time.
3. Implement the narrowest reversible version with typed artifacts, least privilege, and complete logs.
4. Replay representative successes, failures, ambiguous cases, and adversarial inputs.
5. Run in shadow mode or a small canary; inspect every disagreement and regression.
6. Expand only when the benefit persists and downstream review/security/maintenance burden does not rise.

Comment insights

No substantive comments were extracted. That absence is recorded rather than replaced with invented audience consensus.

Comments are distilled as audience evidence only. Agreement, skepticism, memorable phrases, and practitioner additions can identify adoption concerns, but likes and anecdotes do not verify technical claims.

Deep research on the creator’s main claims

Web search was attempted for every topic but the configured provider returned a quota-limit error. To avoid fabricating current findings, this analysis uses stable official documentation, standards, repositories, and established references identifiable from the subject matter:

• Terminal-Bench — external corroboration or limiting context; vendor pages are treated as primary product claims, not independent performance proof.
• Docker documentation — external corroboration or limiting context; vendor pages are treated as primary product claims, not independent performance proof.
• OpenTelemetry traces — external corroboration or limiting context; vendor pages are treated as primary product claims, not independent performance proof.
• Snorkel AI resources — external corroboration or limiting context; vendor pages are treated as primary product claims, not independent performance proof.

Strongest supporting evidence: the sources corroborate the underlying mechanisms—trace instrumentation, layered evaluation, secure development, repeatable containers/benchmarks, pairwise or rubric-based assessment, platform reuse, and operational controls.
Strongest contradicting or limiting evidence: none of these sources proves the speaker’s vendor-specific numbers, exact timelines, or universal superiority. Simulation gaps, evaluator bias, distribution shift, human disagreement, maintenance cost, and organizational economics remain counterevidence to broad claims.
Verified fact versus interpretation: transcript and extracted frames verify what was said and shown. External sources verify that the named patterns/tools exist. Accuracy, causal impact, prevalence, and comparative performance remain interpretations unless backed by a public method and reproducible results.

My verdicts on major claims

1. Every production agent needs a private benchmark

Verdict: Mostly agree — high confidence. Consequential agents need representative repeatable tests; very small prototypes can begin with a lighter suite.

What is over/underclaimed: conference framing is narrower than a reproducible comparative study; confidence is reduced where methods, samples, or raw results are unavailable.

Practical takeaway: validate the mechanism against a simpler baseline with held-out cases, cost/latency, failure analysis, and rollback.

2. A trace is evidence, not an experiment

Verdict: Strongly agree — high confidence. Observational production data does not hold environment and inputs constant.

What is over/underclaimed: conference framing is narrower than a reproducible comparative study; confidence is reduced where methods, samples, or raw results are unavailable.

Practical takeaway: validate the mechanism against a simpler baseline with held-out cases, cost/latency, failure analysis, and rollback.

3. Private benchmarks are more useful for shipping than public benchmarks

Verdict: Agree — high confidence. Public sets orient capability; private tasks represent local tools, policies, costs, and failure modes.

What is over/underclaimed: conference framing is narrower than a reproducible comparative study; confidence is reduced where methods, samples, or raw results are unavailable.

Practical takeaway: validate the mechanism against a simpler baseline with held-out cases, cost/latency, failure analysis, and rollback.

4. Simulation can faithfully reproduce production

Verdict: Mixed — medium confidence. It improves repeatability but users, third-party services, distribution shift, and operational load remain imperfect.

What is over/underclaimed: conference framing is narrower than a reproducible comparative study; confidence is reduced where methods, samples, or raw results are unavailable.

Practical takeaway: validate the mechanism against a simpler baseline with held-out cases, cost/latency, failure analysis, and rollback.

Screen-level insights

• 0:32: The title slide says “From agent traces to agent simulations” and “Building repeatable, production-like benchmarks for AI agents.”
• 1:02: “Three things to take away” visibly structures the claims around benchmarks, production likeness, and lifecycle integration.
• 2:04: “A trace is evidence, not an experiment” pairs a workflow diagram with trace/code detail, making the observational-versus-controlled distinction concrete.
• 7:41: “Anatomy of a benchmark task” shows code/config beside Agent workflow, User workflow, Metadata, and Failure modes/assertions—the most implementation-specific visual.
• 17:21: The transcript at the final frame discusses a held-out train/validation split, supporting leakage controls even though the image itself is speaker-only.

The screen audit was performed from extracted key-frame contact sheets and nearby transcript. Speaker-only frames are explicitly described as such; unreadable UI or hidden details were not inferred.

My read / why it matters

The useful question is not whether the presentation’s preferred agent, evaluator, benchmark, or job model sounds plausible. It is whether the mechanism creates repeatable evidence, catches failures a cheaper baseline misses, preserves security and reversibility, and improves the full lifecycle metric that users actually care about. Adopt the narrow part that survives replay, adversarial testing, and operational review.

Verification notes

Four separate passes were completed before publication:

1. Source/evidence audit: presentation claims, externally corroborated mechanisms, vendor assertions, and unresolved metrics were separated. Direct links and limiting evidence are included.
2. Transcript/comment/frame fidelity audit: timestamps come from extracted caption chunks; comment language remains attributed as opinion; screen descriptions were checked against frame contact sheets and nearby transcript.
3. Hallucination/overclaim audit: exact vendor metrics, causal claims, historical timelines, and universal recommendations were downgraded where public methods or primary data were unavailable. No unverified install command was added.
4. Actionable Insights audit: all five top items name a concrete first move, artifacts or controls, evaluation criteria, and a caution/rollout boundary. Summary-only bullets were rejected.

Residual uncertainty: automated captions may misrecognize names; web search was quota-blocked; linked tools were not executed; several claims are vendor or firsthand reports; and key frames sample rather than exhaust the full video. Recheck primary studies and current product documentation before procurement, policy, or high-risk deployment decisions.