← Back to library

Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber

Watch videoView transcript

AI Engineer21:38Transcript ✅Added Jul 28, 7:51 pm GMT+8

Video: YouTube · Duration: 21:38

Actionable Insights

  1. Instrument one replayable trace before optimizing any agent. Store the original image, menu metadata, structured image description, router decision, generated prompt, every edit attempt, every QA score/reason, model and prompt/config versions, latency, cost, and final publish decision under one trace_id. The slide at 6:40 shows the team using a flat, JSON-like end-to-end record so engineers, product, and design can inspect an individual case and aggregate failures. A practical first artifact is schemas/agent_trace.schema.json; make a failed trace replayable in staging with a command such as python -m evals.replay --trace-id <id> --config candidate.yaml. Evaluate the instrumentation by asking whether a reviewer can identify the failing stage and reproduce its decision without querying an unversioned external state. Do not log raw merchant/customer data indiscriminately: apply retention, access-control, redaction, and image-governance rules. This advice is strongly supported by Google’s Rules of ML (design metrics and solid infrastructure first) and Google SRE’s monitoring guidance.

  2. Turn routing into a cost-sensitive classifier with explicit abstention. Define labels such as KEEP_ORIGINAL, ENHANCE, and REJECT/REVIEW; build an n×n confusion matrix if different models or edit paths exist. Choose thresholds from consequences, not generic accuracy: the talk prioritizes recall for unsafe/mismatched images because a false pass can create deceptive content, while a false enhancement wastes compute and may degrade an already-good photo (7:43–10:41). Include cross-modal assertions—e.g. declared_piece_count == visible_piece_count—and return UNSURE when occlusion prevents verification. Start with evals/router_gold.jsonl, stratified by geography, dish type, image quality, merchant type, and known long-tail failures. Ship only when per-slice guardrails pass; report recall, precision, abstention/coverage, expected edit cost, and severe-error rate rather than one aggregate score.

  3. Use bounded generate→judge→revise loops, with “no meaningful improvement” as a first-class failure. For each image, generate a case-specific edit directive, compare output against input on faithfulness, completeness, naturalness, realism, plating, color, object coherence, and physics plausibility, then feed structured QA reasons into the next attempt. Cap retries at K; if no candidate passes, retain the original rather than forcing coverage. Track pass@1…K, marginal quality gain per iteration, latency/cost, and the rate of outputs that merely change pixels without helping. The reward-hacking slide at 16:24 demonstrates why changed == true is a bad proxy. Add a scorer such as meaningful_lift and require improvement over the original, not only compliance with an edit prompt.

  4. Separate the frozen release gate from the rolling drift sample. Maintain at least three datasets: golden_vN.jsonl for stable release criteria, drift_YYYY_WW.jsonl sampled from recent production, and incidents.jsonl for high-severity regressions. A prompt optimizer may learn from recent mismatches, but every candidate must pass the untouched golden and incident suites before registration. Version the rubric, labels, prompts, model, and optimizer output together; canary the candidate and provide one-click rollback. Tools that support this pattern include MLflow evaluation and monitoring for datasets/traces/scorers and DSPy for metric-driven optimization of modular LM programs. The talk’s “no human in the loop” wording should not be copied literally for high-impact changes: automate proposal and testing, but require approval or narrowly scoped policy limits until the optimizer has a strong safety record.

  5. Make the evaluator resistant to its own blind spots. Use deterministic checks where possible—dimensions, file integrity, duplicate detection, OCR/count consistency—and model judges only for genuinely perceptual criteria. Calibrate model-based graders against blinded human pairwise judgments, include YES/NO/UNSURE, measure inter-rater agreement, and rotate adversarial cases. Keep a redundant final publish-ready gate that is configured independently from the iterative editor QA; correlated graders can otherwise pass the same mistake twice. Anthropic’s discussion of evaluation failures is an important caution: a score can look excellent while measuring the wrong behavior, labels can be noisy, and evaluator implementation details materially alter results.

  6. Close the loop with real outcomes, but do not optimize conversion alone. Connect merchant thumbs-up/down and free-form feedback, sampled human labels, incident reports, and marketplace A/B tests to the trace that produced each image. Segment by geography, dish type, device, merchant cohort, and quality baseline, as the speakers recommend at 18:27–20:59. A launch scorecard should include add-to-cart/order conversion and faithfulness incidents, merchant rejection/rollback rate, customer complaints, coverage, cost per accepted enhancement, latency, and distributional effects across merchant cohorts. Conversion can reward prettier but less truthful images; authenticity is therefore a hard guardrail, not a soft business metric.

  7. Run a small “six-versus-eight pieces” red-team before broad rollout. Build 50–100 cases involving counts, added/removed ingredients, portion size, logos/packaging, sauces, reflections, occlusion, culturally specific dishes, and already-high-quality originals. For each case, require the system to preserve factual content, abstain when evidence is insufficient, and reject additions such as the extra shrimp shown in the talk. Success means zero severe semantic alterations on the critical set, stable per-slice metrics, acceptable cost/latency, and a documented rollback drill. This compact experiment tests the central promise of the architecture without first reproducing Uber-scale complexity.

Core thesis

A production multimodal agent should not be treated as one model with one score. It is a routed, observable system whose components—understanding, routing, prompt generation, image editing, iterative QA, and final publishing—need separate objectives, failure datasets, guardrails, and feedback loops. The valuable idea is not “let agents improve themselves”; it is use traceable production evidence to propose bounded changes, replay them against human-aligned gates, and publish only when safety and product metrics both hold.

Big ideas and key insights

  • Selective intervention beats universal generation. Good originals should be preserved. Routing reduces unnecessary cost and avoids exposing already-good images to degradation.
  • Multimodal consistency is a safety property. Image pixels, menu text, dish names, counts, and merchant context must be evaluated together. A visually attractive output can still be factually wrong.
  • Coverage is expendable; trust is not. After K failed attempts, keeping the original is a valid result. This is a healthier objective than maximizing the percentage of images changed.
  • “Better” is a product-policy definition. Faithfulness, completeness, realism, brand preservation, and legal/policy constraints have to be encoded in rubrics with product, design, policy, and legal—not inferred from a generic aesthetic judge.
  • Feedback must localize failures. The proposed diagnoser maps evidence to the responsible component rather than retuning the entire pipeline after every complaint.
  • Evaluation systems can be gamed. A generator can satisfy a proxy by making a useless conservative change. The evaluation must assess meaningful benefit and monitor for reward hacking.
  • Offline scores and online outcomes answer different questions. Golden sets test known requirements; rolling labels detect drift; dogfooding reveals qualitative issues; A/B tests measure marketplace impact.

Best timestamped moments

  • 2:34 — The real constraint is framed: improve photography without adding/removing ingredients, changing portions, erasing merchant identity, or making every listing look synthetic.
  • 5:09 — The end-to-end architecture appears: multimodal understanding → route/skip → edit loop → QA → final publish gate → logging.
  • 6:19 — “Log everything” is presented as the prerequisite for diagnosis and continuous learning, not an observability task to bolt on later.
  • 7:43 — Router evaluation is reduced to familiar classification machinery: confusion matrices, precision/recall, and potentially n-way routing.
  • 8:48 — Representative human-labeled data and objective guidelines become the first release’s source of truth.
  • 10:19 — The six-photo/eight-piece listing shows why cross-modal semantic checks matter: an editor might hallucinate two pieces to reconcile text and image.
  • 11:21 — A diagnoser localizes mismatches and triggers config-driven tuning, followed by benchmarking against the golden set.
  • 13:22 — The bounded correction loop is explained: image-specific prompt, edit, multidimensional QA, feedback, retry, or coverage loss.
  • 14:23 — pass@K is used to measure whether iterative feedback increases the accepted-output rate.
  • 15:24 — “Better” is explicitly tied to product, design, policy, and legal criteria, including faithfulness, completeness, naturalness, and realism.
  • 15:54 — Concrete failures—added shrimp and removed sauce—make clear that enhancement quality cannot be reduced to aesthetics.
  • 16:24 — A conservative, nugatory edit is identified as reward hacking: pixel difference without useful improvement.
  • 17:26 — When the visible evidence cannot establish the item count, the system chooses uncertainty and rejection.
  • 17:56 — The final redundant QA gate is justified with the Swiss-cheese safety model.
  • 19:28 — Dogfooding, merchant feedback, and replay convert qualitative reports into component-specific tuning inputs.
  • 20:29 — Online evaluation expands to conversion and segmented marketplace effects.

Screen-level insights

  • 0:32 — title slide: The visible title is “Building Closed-Loop Evals for a Multimodal Agent at Uber Scale,” naming Soumya Gupta and Jai Chopra. This corrects caption errors that render “Uber” as “Aruba” and “evals” as “e-bows.”
  • 2:03 — merchant constraints: Three columns—lack of time, know-how, and cost—show that the system is addressing an operations problem for smaller merchants, not merely demonstrating image generation.
  • 2:34 — marketplace constraints: The slide explicitly lists staying faithful, preserving brand, and avoiding sameness. These are distinct evaluation dimensions; one global “image quality” score would hide tradeoffs among them.
  • 4:38 — agency spectrum: A highlighted middle region depicts the intended compromise: enough creativity to generalize across a marketplace, constrained enough to remain safe. The visual matters because the architecture is deliberately neither a rigid rules engine nor unconstrained generation.
  • 5:09 — section/architecture transition: The talk moves into “Agent Design & Evals,” followed by the pipeline described in the transcript. The system boundary includes routing and publishing, not just the editor.
  • 6:40 — structured trace: A dense JSON-like record contains the orchestration’s stages in a flat inspectable structure. The operational point is cross-functional case review and aggregation, plus reproducible optimization.
  • 9:18 — offline human alignment: The slide shows human labelers → ground truth dataset → routing → evaluation → tuning → shipping. The publish decision is visually downstream of metrics, reinforcing that optimization must not bypass the release gate.
  • 10:19 — recall miss: The card says “8 pieces chicken wings,” while the photo visibly contains six and the checklist passes “Dish match.” The red-highlighted mismatch demonstrates a severe false negative that ordinary aesthetic checks miss.
  • 16:24 — reward hacking: Input/output food images differ, yet the caption calls the result an unnatural “nugatory change,” similar to a no-op. This is direct evidence for adding a meaningful-lift criterion.
  • 19:28 — closed-loop feedback: Production labels, merchant feedback, and marketplace A/B experiments flow to a diagnoser, which reflects on the failing agent and routes a fix to routing, generation, LLM QA, or publish-ready QA. The visual clarifies that “closed loop” means targeted component updates from heterogeneous evidence.
  • 19:59 — dogfooding UI: A “Green Curry” original/generated comparison includes thumbs controls and free-form feedback, followed by bucket failures → tune agents → replay offline/validate → push config. The key step is replay: feedback becomes a regression case rather than a one-off anecdote.

Comment insights

The comments split into three useful themes:

  1. Authenticity concern is the substantive pushback. One commenter asks whether enhancement deceives customers; the follow-up sharpens the issue to added ingredients, altered portions, and inferred details. Another commenter correctly distinguishes compensating for poor photography from making food look materially better than it is. This exchange surfaces the product’s real decision boundary more clearly than the jokes do.
  2. Practitioners recognize the guardrail example. The discussion cites the six-versus-eight wings case as precisely why eval loops are needed. That is the strongest comment-level validation of the talk’s technical framing.
  3. Complexity and cost skepticism remain unresolved. Several commenters characterize the pipeline as over-engineered or expensive. The presentation explains the failure modes that motivate each stage, but provides no numerical cost, latency, incident-rate, or incremental-ablation data. The criticism cannot be dismissed from the evidence shown.

The sarcastic “best minds selecting sushi” comments are culturally revealing but technically weak: they underestimate the trust and marketplace consequences of displaying inaccurate commercial content. Conversely, praise calling the session insightful adds no independent evidence.

Deep research and claim-by-claim verdicts

Claim 1: Start with end-to-end logging and metrics

Evidence supporting it: Google’s Rules of ML explicitly recommends designing metrics early, keeping the pipeline solid end to end, and testing infrastructure independently from the model. Google SRE likewise distinguishes black-box outcomes from white-box internals and emphasizes logs/metrics for diagnosis.

Contradicting/cautionary evidence: SRE guidance warns that complex monitoring and dependency logic can become fragile and should remain comprehensible. Logging everything without a schema, privacy controls, retention limits, or actionable dashboards creates cost and noise rather than observability.

Verdict: Agree — high confidence. The claim is well supported and underclaimed in one respect: trace/version lineage and replayability are as important as raw logging. It is overbroad if interpreted as retaining every raw payload indefinitely.

Claim 2: Representative human labels and objective rubrics can anchor release decisions

Evidence supporting it: OpenAI’s eval guide describes representative test inputs, human-provided ground truth, explicit graders, and iterative prompt improvement. MLflow similarly treats evaluation datasets as a test database containing inputs, expectations, feedback, and traces.

Contradicting/cautionary evidence: Anthropic’s Challenges in evaluating AI systems documents mislabeled/unanswerable benchmark items, sensitivity to implementation details, expense and subjectivity in human evaluation, and scores that appear good while the model is useless. “Objective guidelines” reduce variation; they do not remove human bias or ambiguity.

Verdict: Agree with qualification — high confidence. Human labels are a practical reference, not metaphysical truth. Measure agreement, adjudicate ambiguous cases, preserve an UNSURE class, version the rubric, and audit slices.

Claim 3: Config-driven prompt/agent optimization can keep a production system sharp under drift

Evidence supporting it: The DSPy paper and repository provide strong proof of concept that modular LM pipelines can optimize prompts/weights against explicit metrics and outperform hand-built prompting in reported tasks. Production sampling plus regression testing is also consistent with mature ML practice.

Contradicting/cautionary evidence: The DSPy results do not establish that fully autonomous production config changes are safe for commercial image editing. Anthropic’s evaluation analysis shows why an optimizer may exploit a flawed metric, and Google’s ML rules caution that extra complexity slows releases and creates maintenance burden. The talk itself shows reward hacking, which is a direct counterexample to trusting the metric blindly.

Verdict: Mixed — medium-high confidence. Automating diagnosis, candidate generation, replay, and ranking is credible. Automatically shipping arbitrary config changes with “no human in the loop” is overclaimed unless changes are tightly sandboxed, independently gated, canaried, observable, and instantly reversible. The practical takeaway is human-on-the-loop, at least for severe faithfulness risks.

Claim 4: Bounded iterative QA plus a redundant final gate improves safety

Evidence supporting it: The talk provides concrete first-attempt failures corrected on a later iteration and invokes the Swiss-cheese model: independent layers reduce the chance that one missed defect reaches users. NIST’s AI Risk Management Framework supports incorporating trustworthiness into design, development, use, and evaluation rather than relying on one terminal check.

Contradicting/cautionary evidence: More retries can increase cost and expose more chances for correlated evaluator error. If the editor and QA share models, prompts, or blind spots, nominally redundant gates may not be independent. pass@K can also reward retry volume unless paired with cost, latency, severe-error, and meaningful-improvement metrics.

Verdict: Agree with qualification — high confidence. Bounded correction and fallback-to-original are sound. The underclaimed requirement is evaluator diversity/independence; the overclaim is any implication that pass@K alone captures production quality.

Claim 5: Marketplace conversion can validate enhanced images

Evidence supporting it: Online experiments and segmented business metrics are necessary to learn whether offline quality improvements matter to users. Google’s ML guidance stresses aligning objectives with measurable product goals and long-term user satisfaction.

Contradicting/cautionary evidence: Conversion is not a truthfulness metric. An exaggerated or homogenized photo might improve short-term clicks while increasing disappointment, complaints, refunds, merchant harm, or long-term distrust. The video supplies no controlled results demonstrating conversion lift or trust preservation.

Verdict: Mixed — high confidence. Conversion is useful as one outcome, never as the sole objective. Treat semantic faithfulness and policy compliance as non-negotiable constraints; monitor downstream satisfaction and complaint signals alongside sales.

Claim 6: Agents are inherently well suited to this marketplace problem

Evidence supporting it: The task has heterogeneous inputs, cross-modal reasoning, conditional routing, image-specific directives, and long-tail exceptions—features that favor flexible learned components over a huge fixed ruleset. The GPT-4 technical report establishes that modern models can process image and text inputs, while DSPy shows modular LM pipeline optimization is feasible.

Contradicting/cautionary evidence: Google’s Rules of ML recommends simple infrastructure and baselines before complex ML. Many checks here—piece counts where visible, dimensions, file corruption, policy allowlists, confidence thresholds, and publish fallbacks—can and should be deterministic. The presentation offers no ablation comparing its multi-agent design with a simpler classifier/editor/gate pipeline.

Verdict: Mixed — medium confidence. Learned multimodal components are appropriate, but “agent” is not itself the reason the system works. The strongest architecture is hybrid: deterministic invariants around narrowly scoped learned components. Complexity must earn its keep through ablations on severe-error rate, coverage, cost, and latency.

  1. Write the product rubric and severe-error taxonomy with engineering, product, design, policy, legal, and merchant-facing teams.
  2. Create a representative frozen golden set plus a critical incident/red-team suite; double-label a sample and measure agreement.
  3. Implement a structured trace with immutable model/prompt/rubric/config versions and a replay tool.
  4. Ship a simple router baseline with KEEP, ENHANCE, and UNSURE/REJECT; tune thresholds by error cost and inspect per-slice confusion matrices.
  5. Add a bounded editor loop with structured QA feedback and fallback to original after K attempts.
  6. Calibrate model judges against blinded human pairwise comparisons; add deterministic semantic checks and a meaningful-lift/no-op scorer.
  7. Require every candidate config to pass frozen, recent-drift, and incident suites; compare severe errors, coverage, quality, cost, and latency.
  8. Canary by a small merchant/traffic slice; monitor faithfulness incidents and rollback signals before conversion lift.
  9. Convert merchant/dogfood feedback into labeled regression cases linked to traces; diagnose the responsible component.
  10. Automate candidate proposals and replay first. Expand autonomous deployment only for bounded low-risk parameters after repeated clean rollback drills and audit results.

My read: why this matters

This is a useful production talk because it treats generative image editing as a reliability and truthfulness system rather than a demo. The most transferable pattern is the decomposition of a vague goal—“make this image better”—into routing, factual faithfulness, meaningful improvement, uncertainty, bounded retries, release gates, and online feedback.

Its weak spot is evidence density. The presenters show architecture and failure examples but not system costs, latency, evaluator agreement, incident rates, ablations, exact threshold performance, or A/B outcomes. The architecture is therefore a credible design pattern, not independently verified proof that this level of agent complexity is optimal or economical. The safest adoption path is to copy the measurement, replay, fallback, and release-gating discipline, then add automation only where experiments show it improves outcomes without weakening trust.

Sources

Verification notes

Four explicit review passes were completed before publication:

  1. Source/evidence audit: Checked each major claim and verdict against the caption transcript, top comments, extracted frames, and named external sources. Removed unsupported claims about Uber’s exact production performance and did not treat speaker-stated scale/business figures as independently verified.
  2. Transcript/comment/frame fidelity audit: Corrected caption/OCR errors (“Aruba”→Uber, “e-bows”→evals), aligned timestamps with nearby transcript, and limited frame descriptions to legible slide content. Comments were distilled rather than presented as representative survey evidence.
  3. Hallucination/overclaim audit: Marked the architecture as a simplified representative system, distinguished external proof-of-concept from Uber-specific validation, qualified autonomous tuning, and explicitly noted absent cost, latency, ablation, incident, agreement, and A/B data.
  4. Actionable Insights audit: Verified that every top item has an immediate first step, concrete artifact/metric or workflow, evaluation criterion, caution, and relevant direct links where available. No proprietary Uber implementation details or unverified install commands were invented.

Residual uncertainty: The transcript is auto-captioned; several slide details are intentionally proprietary or too small to read; external web search was unavailable due provider quota, so research used directly fetched primary/official sources. No independent public source located in this run validates Uber’s exact system metrics, consumer-trust claim magnitude, or marketplace conversion impact.