Your Coding Agent Should Do AI System Engineering — Ben Burtenshaw, Hugging Face

Processed: 2026-05-25 10:18 UTC

Actionable Insights

• Turn specialist engineering workflows into agent skills before asking for expert-level output. For CUDA/kernel work, start from Hugging Face’s kernels skill rather than a blank prompt: pip install git+https://github.com/huggingface/kernels.git#subdirectory=kernels, then kernels skills add cuda-kernels --claude, --codex, or --opencode depending on your harness. Use it for tasks like: “Build a vectorized RMSNorm kernel for H100 targeting Qwen3-8B; include PyTorch bindings, microbenchmarks, and end-to-end benchmark.” Evaluate success by correctness against PyTorch, latency/speedup, hardware compatibility, and whether torch.compile/integration paths still work. Caution: generated kernels are high-risk systems code; require hardware-specific tests and numerical tolerances before production.

• Package acceleration work as distributable artifacts, not one-off patches. The talk’s core operational move is to make kernels repo-like: metadata for hardware, CUDA/PyTorch/Python compatibility, examples, and benchmark scripts. If you write or agent-generate a kernel, create a small “kernel card” with target model/operator, supported GPU architecture, dtype, install command, benchmark method, baseline, speedup, and failure cases. The relevant ecosystem links are Hugging Face Kernel Hub / kernels (https://huggingface.co/blog/hello-hf-kernels, https://github.com/huggingface/kernels) and the CUDA agent-skill post (https://huggingface.co/blog/custom-cuda-kernels-agent-skills). This prevents the common failure mode where an impressive kernel cannot be reproduced, installed, or routed into inference.

• Aim kernel experiments at memory-bound operators first. Ben’s “compute, memory, overhead” framing and the efficiency slide matter because a random CUDA rewrite is not automatically useful. Pick operations where profiling shows memory bandwidth, dispatch overhead, or tensor movement dominates; compare against known wins such as FlashAttention-style arithmetic-intensity improvements. First step: profile with PyTorch profiler/Nsight, identify top memory-bound ops, then ask the agent for a bounded kernel plus benchmark. Evaluation criteria: lower wall-clock latency, stable output error, no regression across batch/sequence sizes, and clear improvement over PyTorch baseline on the target GPU.

• Build an evals/ directory for every skill you depend on. The upskill portion claims skills can be evaluated across models; make that concrete by storing the same task suite under evals/<skill-name>/ with fixtures, expected outputs, runtime constraints, and cost/token measurements. Run it whenever you change the skill or switch model providers. Track not only pass/fail but also token count, runtime, tool calls, and repair attempts. This makes “Kimi is cheaper here” or “Haiku is reliable enough” a measured decision rather than a vibe.

• Use multi-agent auto-research only where experiments are verifiable. The AutoLab section is most useful for tasks with a scalar metric: validation BPC, benchmark latency, accuracy, throughput, or cost. A safe workflow is: researcher proposes paper-derived hypotheses, planner dedupes and queues, worker implements isolated branches/jobs, reviewer rejects stale/invalid ideas, and reporter writes a dashboard from open logs. Try this on nanoGPT/nanochat-style training scripts or kernel benchmarks before applying it to product code. Caution: branch sprawl, duplicate hypotheses, hidden failed runs, and metric gaming are the main integration risks.

Core thesis

Coding agents can move beyond CRUD/product-code tasks into AI systems engineering — CUDA kernels, fine-tuning, and automated research — but only when the ecosystem exposes standard, open, reproducible primitives: skills, benchmark scripts, hub repositories, compatibility metadata, jobs, and tracking.

Big ideas / key insights

• Agents are becoming viable for low-level AI performance work. At 2:11, Ben points to GPU Mode hackathons, AMD hackathons, and KernelBench as evidence that agents can write valid/optimized kernels. The stronger claim is not “agents replace kernel engineers,” but “agents can contribute when given domain patterns and measurement loops.”
• Distribution is as important as generation. The 5:17–5:48 segment argues that a kernel must be installable and hardware-compatible, not just impressive in a local notebook.
• Skills convert zero-shot into few-shot. At 6:18–7:20, the talk shows skills as simple file-based context (SKILL.md, scripts, references, assets), maintained inside projects so agents can open them only when needed.
• Evals close the loop. The upskill section at 8:20–8:50 shifts from “we made a skill” to “we can compare models on the same skill.”
• Multi-agent research needs an open data layer. AutoLab uses branches, job queues, HF Jobs, and Trackio so agents can coordinate through inspectable state rather than hidden chat context.

Best timestamped moments with interpretation

• 2:11 — The talk names KernelBench and GPU hackathons as the first evidence base. Interpretation: benchmarked, constrained tasks are the right place to trust agents, not open-ended magic.
• 4:15 — The efficiency slide separates compute, memory, and overhead. Interpretation: this is the profiling checklist for deciding whether custom kernels are worth writing.
• 5:17 — Kernel distribution via Hub-style repositories. Interpretation: agent output becomes valuable when packaged for reuse.
• 6:18 — Skills shown as a folder structure and markdown instructions. Interpretation: the mechanism is intentionally boring; the value is maintained context plus examples.
• 8:50 — Upskill/eval comparison across models. Interpretation: skill quality should be measured with model-specific cost/reliability tradeoffs.
• 10:20–12:55 — AutoLab roles: researcher, planner, workers, reporter. Interpretation: multi-agent setups should be organized like a lab workflow, not a swarm of unsupervised terminals.

Practical takeaways / recommended workflow

1. Profile before optimizing: identify memory-bound or overhead-heavy operators.
2. Install or create a skill for the domain. For kernels, start with Hugging Face kernels and cuda-kernels skill.
3. Ask for a bounded deliverable: one operator, one GPU target, one model target, numerical tests, and benchmarks.
4. Package the result as a repo/hub artifact with compatibility metadata and a runnable benchmark.
5. Run evals across at least two model/harness combinations if the skill will be reused.
6. Only escalate to multi-agent AutoLab when the objective has a metric and isolated branches/jobs can be cleaned up.

Comment insights

The comment section is small but useful. One viewer praised the camera/editing because slides were readable, which matters here: this talk’s technical value depends heavily on seeing the skill structure, benchmark charts, and dashboard screenshots. Another commenter suggested multiple-token prediction or a second smaller model in the GPU buffer to improve compute utilization in one memory-transfer window; that aligns with the talk’s “keep GPUs warm” theme, though it is a separate inference-architecture idea rather than a direct kernel-distribution solution. Other comments are mostly enthusiasm, not technical pushback.

Deep research on the creator’s main claims

Claim 1: Agents can write production-useful CUDA kernels when given domain skills.

Supporting evidence: Hugging Face’s “Custom Kernels for All from Codex and Claude” says they built an agent skill for production CUDA kernels and had Claude/Codex create working kernels for diffusers and transformers with PyTorch bindings and benchmarks. The post says the skill includes architecture-aware guidance for H100/A100/T4, integration patterns, templates, benchmarking workflows, and Kernel Hub integration. KernelBench and GPU Mode-style competitions also support the narrower statement that LLM agents can solve constrained kernel-generation tasks.

Contradicting/cautionary evidence: CUDA optimization is architecture-specific and correctness-sensitive; the same Hugging Face post lists “brutal surface area” issues: GPU generation differences, diffusers/transformers integration pitfalls, and CUDA/PyTorch/Python version matrices. That supports the need for review, not blind deployment.

Claim 2: Memory, not compute, is often the bottleneck for inference operators.

Supporting evidence: This is broadly consistent with GPU performance engineering and modern LLM serving literature. NVIDIA’s Dynamo KV-cache article states that KV cache grows with prompt length and GPU memory is limited/costly, forcing tradeoffs around eviction, context caps, or more GPUs. Cerebras’ disaggregated inference explainer similarly distinguishes compute-heavy prefill from memory-bandwidth-bound decode.

Contradicting/cautionary evidence: “Memory is usually the bottleneck” is workload-dependent. Dense matrix multiplication, small kernels dominated by launch overhead, and poorly batched workloads can be compute- or overhead-bound. Profiling is required.

Claim 3: Skills and open primitives are better than opaque APIs for agentic engineering.

Supporting evidence: The talk’s file-based skill structure is supported by Hugging Face’s skill blog: roughly 550 tokens of instructions plus scripts, references, troubleshooting docs, and examples. This is directly inspectable and versionable. The practical advantage is that agents can read only relevant context and developers can diff the guidance.

Contradicting/cautionary evidence: Skills can become stale or misleading if project maintainers do not own them. The talk addresses this by preferring project-managed skills over “YOLO” external skills.

Claim 4: Multi-agent automated research can improve training scripts.

Supporting evidence: Andrej Karpathy’s nanoGPT/nanochat auto-research experiments and Ben’s AutoLab pattern support the feasibility of iterative, metric-driven experimentation. The screenshoted validation-BPC curve gives visual evidence of a metric improving across experiments.

Contradicting/cautionary evidence: The evidence is strongest for small, measurable research loops. It does not prove general research autonomy or scientific novelty. Without duplicate detection, reviewer gates, and experiment provenance, multi-agent research can waste compute or overfit to narrow metrics.

Verdicts on major claims

• Agents should do AI systems engineering: Agree, medium-high confidence. The claim is practical when the task has tests, benchmarks, and domain skills. It is overclaimed if read as “agents can autonomously replace systems engineers.” Practical takeaway: use agents as accelerated implementers inside a profiling/test harness.
• Standard repos on the Hub are necessary: Agree, high confidence. Distribution, compatibility, and reproducibility are the difference between a demo and an engineering asset. Practical takeaway: require metadata and benchmarks before adopting a generated kernel.
• Memory is usually the inference bottleneck: Mixed, medium confidence. Often true for LLM decode/KV-cache-heavy workloads and many kernels, but not universal. Practical takeaway: profile first; do not optimize from slogans.
• Skills make agents meaningfully better: Agree, medium confidence. The mechanism is plausible and supported by Hugging Face’s skill/eval work, but gains will vary by model and task. Practical takeaway: treat skills as code: version, test, and evaluate them.
• Multi-agent AutoLab is ready for serious use: Mixed, medium confidence. Ready for bounded experiments with metrics and compute budgets; not ready as an unsupervised research lab. Practical takeaway: begin with one repo, one metric, branch isolation, and a reporter dashboard.

Screen-level insights

• 0:07 — Sponsor/title card, not technical evidence. It establishes event context only.
• 2:11 — A meme slide says “CLAUDE CODE DOES NOT WRITE CUDA KERNELS.” This matters because the talk is explicitly challenging that folk belief with benchmarked examples.
• 4:15 — The “Efficiency in Deep Learning” slide lists compute, memory, and overhead. This visual turns the transcript’s performance discussion into a concrete diagnostic taxonomy.
• 5:17 — “Unified tooling to build compute kernels” shows reproducibility, PyTorch compatibility, and community sharing. This supports the claim that tooling/distribution is the missing layer.
• 5:48 — A Kernel Hub-style page shows install commands, code usage, and supported hardware. This connects “kernel publisher” to an actual user workflow.
• 6:18–7:20 — The “WTF r skills?” slides show SKILL.md, scripts/, references/, assets/, and CLI install commands such as kernels skills add. This is the strongest visual evidence for the file-based skills mechanism.
• 8:50 — “Agents Teaching Agents / upskill” chart compares baseline vs with-skill performance. The visual matters because it frames skills as measurable, not just instructional.
• 10:20–10:52 — AutoLab transition and Karpathy autotune graph show the multi-agent/autoresearch section is built around measured experiment progress.

My read / why it matters

This is a strong “make agents useful by making systems inspectable” talk. The important move is not that agents can write CUDA; it is that the surrounding engineering shape — skills, evals, kernel metadata, hub distribution, Trackio dashboards, HF Jobs — makes agent output reviewable and reusable. For Kx’s OpenClaw/workflow world, the direct lesson is: every recurring technical workflow should have a skill, fixtures, an eval loop, and a publishable artifact.

Verification notes

Four review passes were applied before publishing. Source/evidence audit: checked Hugging Face’s CUDA-kernel skill post, Kernel Hub references, and external inference/KV-cache sources from NVIDIA and Cerebras; claims are limited to what those sources and the transcript support. Transcript/comment/frame fidelity audit: timestamp claims were matched to the extracted transcript and visual frame analysis; the tiny comment set is described as small and not over-weighted. Hallucination/overclaim audit: softened broad claims around “agents can write kernels,” “memory is usually the bottleneck,” and AutoLab readiness; residual uncertainty remains around the exact 94% speedup because the transcript states it but the specific benchmark artifact was not independently reproduced here. Actionable Insights audit: top bullets include concrete commands, links, first steps, evaluation criteria, and cautions; they are tied to video evidence and external sources rather than generic summaries.