Don't Ship Skills Without Evals — Philipp Schmid, Google DeepMind

Actionable Insights

• Add a deterministic eval next to every production skill before rollout. Treat each skill.md like code that needs tests. Create skills/<name>/skill.md, skills/<name>/evals.yaml or evals.json, and a runner such as pytest tests/test_<name>_skill.py or vitest. Include at least trigger tests, happy-path task tests, and regression cases from real failures. Score with deterministic verifiers wherever possible: file exists, API call shape, JSON schema, unit-test pass, or golden diff. Use LLM judges only for subjective quality and calibrate them against human-reviewed examples.

• Audit skill descriptions because they are always paid in context. Philipp emphasizes progressive disclosure: title and description are loaded first, then the body, then references. Keep the description directive-style: “Use this skill when…” and “Do not use when…”. First step: scan all skill descriptions for vague nouns, missing trigger conditions, and overlapping scopes. Evaluate by measuring false triggers and missed triggers on a small prompt suite.

• Split capability skills from preference skills. Capability skills teach the model something it currently cannot do consistently, such as tracing logs or creating a specific app scaffold; they may become obsolete as models improve. Preference skills encode durable company style, workflows, or API usage. Tag each skill with type: capability|preference, owner, expiry review date, and eval coverage. Remove capability skills when a no-skill baseline catches up; protect preference skills with regression tests during model upgrades.

• Keep skill.md lean and move deep references into separate files. The talk cites SkillBench findings that very long skill files can hurt performance and suggests a rough review threshold around 500 lines/words in the transcript. Put cloud-specific or framework-specific detail into refs/aws.md, refs/gcp.md, refs/azure.md, etc., and let the skill route to the relevant reference. Evaluate by comparing task success and token usage before/after the split. Caution: over-splitting can make retrieval brittle; add evals that verify the agent opens the needed reference.

• Prefer user-invoked skills for deterministic developer chores. Philipp distinguishes developer agents from customer-facing agents. For tasks such as creating PRs, staging docs, running release checks, or formatting a migration, make the skill explicitly callable by the engineer rather than hoping the model triggers it. Wrap repeated commands in scripts where possible. Evaluate by time saved and by whether the output is identical across runs.

• Use SkillsBench as a reference, not a substitute for your domain evals. SkillsBench (https://www.skillsbench.ai/) and the paper/repo (https://arxiv.org/abs/2602.12670, https://github.com/benchflow-ai/skillsbench) provide a public framework for testing skill effectiveness across tasks. Start by copying the harness idea: compare no skill vs curated skill vs generated skill. Your production decision should still depend on your own prompts, APIs, and failure modes.

Core thesis

Skills can improve agent performance, but shipping untested skill files is equivalent to merging untested production code. Because agents are non-deterministic, teams need evals to distinguish a bad skill, a bad trigger, and a task the model cannot currently solve.

Big ideas / key insights

• Most developers use coding agents and skills; far fewer maintain evals for those skills.
• Skill design relies on progressive disclosure: short trigger metadata, concise body, deeper references.
• Model-invoked and user-invoked skills require different testing strategies.
• Human-written, curated skills appear stronger than casually AI-generated skills in public benchmark results.
• LLM-as-judge evals can help, but deterministic verifiers are safer for production gates.

Best timestamped moments with interpretation

• 0:31 — The audience hand-raise makes the adoption/testing gap visible: many use skills; few test them.
• 1:33–2:04 — Philipp separates agents engineers use from agents customers use. Engineers can notice a missed skill; customers cannot.
• 3:05–4:05 — Capability vs preference skills is the talk’s best taxonomy for lifecycle management.
• 4:36–5:07 — SkillsBench results are used to argue that skills help on average but low-quality AI-generated skills can hurt.
• 6:11–8:45 — The practical design section: directive descriptions, lean bodies, and layered references.
• 12:50 — The Gemini Interactions API example mentions 117 test cases, showing what a serious skill eval can look like.

Practical takeaways / recommended workflow

1. Inventory every skill, owner, trigger condition, and intended task class.
2. Add a baseline eval: no skill vs current skill.
3. Add trigger/mis-trigger cases for model-invoked skills.
4. Use deterministic verifiers first, LLM judges second.
5. Re-run the suite on skill edits and model upgrades.
6. Retire capability skills when the baseline model no longer needs them.

Comment insights

Commenters mostly agree with the core premise but sharpen it. Several reduce the message to “skills need unit tests,” which is fair but undersells trigger evaluation. One commenter asks why eval cases are YAML/JSON if Python still runs them; this is a useful implementation caveat: declarative cases improve consistency, but native pytest/vitest can be more powerful for complex domains. Another warns that LLM-as-a-judge is unreliable, which aligns with the need for deterministic checks. Practitioners also report real chaos from unverified skills and ask for battle-tested example skill libraries.

Deep research on the creator’s main claims

• SkillsBench exists and is directly relevant. Search results identify SkillsBench as a benchmark for agent skills with tasks across domains, deterministic verifiers, a public website, arXiv paper, and GitHub repository. This supports Philipp’s use of it as evidence that skills can improve performance.
• Skills can help, but generated skills are not automatically safe. The SkillsBench framing compares no-skill, curated-skill, and generated-skill settings. That supports the talk’s warning to test AI-generated skill files rather than trusting them.
• LLM judges require caution. Papers and surveys such as “Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge,” “Reliability without Validity,” and a 2025 ScienceDirect survey report concerns around consistency, bias, order effects, and validity. This contradicts any simplistic “just judge it with another model” approach.
• Progressive disclosure is consistent with agent memory/tool design. Keeping always-loaded instructions short and pushing rare detail into references is consistent with context-window cost and retrieval precision constraints, though the exact 500-line threshold should be treated as a heuristic unless confirmed in the specific benchmark release.

My verdicts on major claims

• Claim: do not ship skills without evals. Verdict: agree, high confidence. Non-determinism makes informal manual testing insufficient.
• Claim: skills improve performance by roughly meaningful margins on average. Verdict: mostly agree, medium confidence. SkillsBench supports skill effectiveness, but averages can hide domain-specific failures.
• Claim: AI-generated skills can hurt performance. Verdict: agree, medium-high confidence. This is plausible and supported by benchmark comparisons, but teams should test against their own workload.
• Claim: LLM-as-judge evals are enough. Verdict: disagree if implied, high confidence. Comments and external literature show reliability problems; use deterministic checks whenever possible.

Screen-level insights

• 0:31 frame — The audience poll visually reinforces the gap between skill adoption and evaluation discipline.
• 1:33 frame — The slide/context names developer tools such as Antigravity, Cursor, and Claude Code; the key visual point is the distinction between supervised developer use and customer-facing autonomy.
• 3:05 frame — The capability/preference distinction appears near the frame, giving teams an immediately usable categorization.
• 4:36 and 5:07 frames — SkillBench and AI-generated skill discussion appears onscreen while Philipp warns that human-written skills perform best and bloated files need review.
• 6:11 frame — User-invoked vs model-invoked skills is connected to real developer chores like PR creation and docs staging.
• 12:50 frame — The Gemini Interactions API example shows an eval suite with 117 cases, making the talk operational rather than purely philosophical.

My read / why it matters

This is the pragmatic companion to the current skill-file boom. The important move is not “write better prompts”; it is to make skill behavior observable, testable, and regressible like software.

Verification notes

Four verification passes completed: source/evidence audit, transcript/comment/frame fidelity audit, hallucination/overclaim audit, and Actionable Insights audit. Actionable items include concrete file layouts, eval types, first steps, metrics, and cautions. External checks covered SkillsBench website/arXiv/GitHub search results and LLM-as-judge reliability literature. Residual uncertainty: the exact SkillBench aggregate percentages and the 500-line/word threshold should be verified against the released benchmark materials before citing as hard limits.