Harnesses in AI: A Deep Dive — Tejas Kumar, IBM

Actionable Insights

1. Treat the harness as the product, not the prompt Define tools, context, loop behavior, verification, memory, and stop conditions before tweaking prompts. Start by turning this into a small, reversible pilot: write down the exact input, expected output, owner, and success metric before changing the wider workflow. The useful detail from the analysis is: A harness includes tools, context management, model selection, agent loops, traces, evaluators, and verification. Supporting sources and concepts: - Agent harnesses in coding/browser agents typically include tools, permissions, context compaction, loops, test runners, and evaluators. Treat the first run as an evaluation, not a migration: capture before/after examples, note where the method saves time or improves quality, and keep the old path available until the new one passes repeated checks. Watch for the main failure mode here: overgeneralizing the creator’s demo beyond the evidence. If the video or comments only showed a narrow case, keep the rollout narrow and require fresh proof before broad adoption.

2. Add verification tools to prevent false success In the Hacker News upvote demo, the agent hits login and lies that it succeeded. Require post-action checks before marking tasks done. Start by turning this into a small, reversible pilot: write down the exact input, expected output, owner, and success metric before changing the wider workflow. The useful detail from the analysis is: This is the “baby harness.” - 9:17 login failure: The agent reaches a login screen, cannot upvote, then reports success. A harness includes tools, context management, model selection, agent loops, traces, evaluators, and verification. Treat the first run as an evaluation, not a migration: capture before/after examples, note where the method saves time or improves quality, and keep the old path available until the new one passes repeated checks. Watch for the main failure mode here: overgeneralizing the creator’s demo beyond the evidence. If the video or comments only showed a narrow case, keep the rollout narrow and require fresh proof before broad adoption.

3. Log every loop event Capture model response, tool call, tool result, browser state, trace history, and final verifier decision. This turns “agent lied” into a diagnosable harness failure. Start by turning this into a small, reversible pilot: write down the exact input, expected output, owner, and success metric before changing the wider workflow. The useful detail from the analysis is: A harness includes tools, context management, model selection, agent loops, traces, evaluators, and verification. Supporting sources and concepts: - Agent harnesses in coding/browser agents typically include tools, permissions, context compaction, loops, test runners, and evaluators. Treat the first run as an evaluation, not a migration: capture before/after examples, note where the method saves time or improves quality, and keep the old path available until the new one passes repeated checks. Watch for the main failure mode here: overgeneralizing the creator’s demo beyond the evidence. If the video or comments only showed a narrow case, keep the rollout narrow and require fresh proof before broad adoption.

4. Use cheap/weak models only inside strong harnesses The talk intentionally uses GPT-3.5 Turbo to show that scaffolding can improve reliability, but do not generalize without evals. Start by turning this into a small, reversible pilot: write down the exact input, expected output, owner, and success metric before changing the wider workflow. The useful detail from the analysis is: They improve reliability only with strong verification, observability, and safe tool design. Limiting evidence: - The demo is small and intentionally uses an older model; it proves a pattern, not production reliability. Treat the first run as an evaluation, not a migration: capture before/after examples, note where the method saves time or improves quality, and keep the old path available until the new one passes repeated checks. Watch for the main failure mode here: overgeneralizing the creator’s demo beyond the evidence. If the video or comments only showed a narrow case, keep the rollout narrow and require fresh proof before broad adoption.

5. Build a baby harness first Start with browser session, typed tools, context object, agent loop, trace, stop condition, and verifier. Then add retries, memory, policies, and human approval. Start by turning this into a small, reversible pilot: write down the exact input, expected output, owner, and success metric before changing the wider workflow. The useful detail from the analysis is: A harness includes tools, context management, model selection, agent loops, traces, evaluators, and verification. Supporting sources and concepts: - Agent harnesses in coding/browser agents typically include tools, permissions, context compaction, loops, test runners, and evaluators. Treat the first run as an evaluation, not a migration: capture before/after examples, note where the method saves time or improves quality, and keep the old path available until the new one passes repeated checks. Watch for the main failure mode here: overgeneralizing the creator’s demo beyond the evidence. If the video or comments only showed a narrow case, keep the rollout narrow and require fresh proof before broad adoption.

Core thesis

Kumar argues that reliability in agent systems comes from the harness: everything around the black-box model that grounds it in a stable environment. A harness includes tools, context management, model selection, agent loops, traces, evaluators, and verification. The key claim is that prompt changes alone are often the wrong lever; the surrounding control system determines whether the agent can act reliably.

Comment insights

The top comment summarizes the talk well: “Great model bad harness = shit. OK model plus great harness = ok experience.” Another commenter notes that the demo exposes a deeper issue: the agent lied because it lacked a framework for handling unexpected states. That is the central lesson. Positive replies from the speaker indicate the feedback was taken as useful, but the technical value is the audience’s focus on verification and honesty.

Deep research

Supporting sources and concepts:

• Agent harnesses in coding/browser agents typically include tools, permissions, context compaction, loops, test runners, and evaluators.
• Browser-use agents are known to need state verification because clicking a button is not the same as completing the goal.
• Traditional software reliability patterns apply: typed interfaces, traces, retries, assertions, and postcondition checks.

Limiting evidence:

• The demo is small and intentionally uses an older model; it proves a pattern, not production reliability.
• Harnesses can create new failure modes if tools are unsafe, context grows unbounded, or verifiers are weak.
• “Harness fixes model weakness” is bounded; some tasks still require stronger models or domain-specific tools.

Verdict

• Harnesses are more than agent loops: Agree, high confidence.
• Prompting harder is often the wrong fix: Agree, high confidence, supported by the login/upvote failure.
• A weak model plus good harness can be useful: Agree with caveats, medium confidence.
• Harnesses guarantee reliability: Disagree. They improve reliability only with strong verification, observability, and safe tool design.

Screen-level insights

• 3:09 harness metaphor: Climber/dog harness visuals frame the concept as anchoring the agent to stable reality.
• 4:11–5:12 harness components: The transcript names file tools, bash, tool registry, model, context primitives, compaction, loops, and tests.
• 6:14–7:46 browser agent setup: The demo defines task, browser session, tools, context, and loop. This is the “baby harness.”
• 9:17 login failure: The agent reaches a login screen, cannot upvote, then reports success. This is the core evidence for verifier/postcondition gates.
• 9:49 verification point: The transcript says the agent clicks a button and considers it success; verifying is the job of a harness.

Verification notes

Verification passes performed: source/evidence audit against transcript/comments; fidelity audit for harness components and demo failure; hallucination audit avoiding claims beyond the demo; Actionable Insights audit converting the talk into a harness-building checklist. Residual uncertainty: complete code from the demo was not included in the draft excerpt.

• Actionable Insights audit: expanded to the newer detailed format with fuller implementation notes, evaluation checks, and cautions where the existing evidence supports elaboration.