← Back to library

How Building with AI Can Double the Throughput of Your Engineering Team — Brian Scanlan, Intercom

Watch videoView transcript

AI Engineer21m 48sTranscript ✅Added May 18, 2:40 pm GMT+8

Creator/speaker: Brian Scanlan, Intercom
Duration: 21:48
Evidence used: extracted transcript/comments, key frames, and external sources listed below.

Actionable Insights

  1. For technical leaders and senior ICs, the useful takeaway is not “buy Claude Code and prod. uctivity doubles.” It is: treat coding agents as an internal engineering platform, with onboarding, skills, hooks, evals, telemetry, and a feedback loop. Start by turning this into a small, reversible pilot: write down the exact input, expected output, owner, and success metric before changing the wider workflow. The useful detail from the analysis is: This supports the feasibility of treating Claude Code as a platform surface. - A positive commenter from Bitloops agrees with durable, testable skills but argues context should be auto-generated and queryable, not hand-maintained markdown. Treat the first run as an evaluation, not a migration: capture before/after examples, note where the method saves time or improves quality, and keep the old path available until the new one passes repeated checks. Watch for the main failure mode here: overgeneralizing the creator’s demo beyond the evidence. If the video or comments only showed a narrow case, keep the rollout narrow and require fresh proof before broad adoption.

Workflow items to implement now

  1. Pick one default agent surface for serious platform work.

    • Candidate: Claude Code — https://docs.anthropic.com/en/docs/claude-code/overview
    • Avoid starting with five unrelated agent harnesses unless you have a clear interoperability reason. Intercom’s argument is that platform compounding matters more than perfect model shopping.
    • Evaluation: adoption rate, task completion rate, PR cycle time, defect escape rate, developer satisfaction.

  2. Turn repeated tribal knowledge into Agent Skills.

    • Skill docs: https://docs.anthropic.com/en/docs/claude-code/skills
    • Create repo-local skills for: flaky test repair, API conventions, security incident triage, migration recipes, frontend component patterns, release checklists.
    • Checklist:
      • Each skill has a concrete trigger and expected output.
      • It includes examples from your codebase.
      • It is small enough to test and revise.
      • It links to source-of-truth docs rather than duplicating stale policies.
      • It has at least one backtest or golden example.

  3. Add lifecycle hooks and guardrails before pushing autonomy.

  4. Instrument the agent platform like production software.

    • Use OpenTelemetry-style traces/logs/metrics where possible: https://opentelemetry.io/docs/what-is-opentelemetry/
    • Capture: prompt/task category, tools used, skills loaded, test results, PR review outcomes, rollback/incident linkage, token/cost, human intervention points.
    • Experiment: compare agent-assisted PRs vs normal PRs on lead time, review churn, escaped defects, and customer-impacting outcomes.

  5. Create “safe PR shape” rules.

  6. Do not use PR count alone as the productivity KPI.

    • Commenters correctly challenged “double PRs” as gameable.
    • Better scorecard: shipped customer outcomes, cycle time, incident rate, rework rate, review latency, support tickets, NPS/CSAT, cost per shipped change.

Commands/tools to try

# Install Claude Code on macOS/Linux/WSL per Anthropic docs
curl -fsSL https://claude.ai/install.sh | bash

# Start in a repo
cd your-repo
claude

Also evaluate GitHub automation if your team wants PR review/triage workflows: https://github.com/anthropics/claude-code-action

Integration cautions

  • Do not copy Intercom’s autonomy level before you have audit logs, permission boundaries, CI quality, and incident linkage.
  • SOC 2/ISO/HIPAA compatibility is plausible, but the transcript only gives Intercom’s claim; your auditor still needs your actual controls.
  • A “single platform” strategy creates lock-in and outage/vendor risk. Keep artifacts portable: markdown skills, tests, scripts, and telemetry schemas.

Core thesis

Scanlan argues that Intercom doubled engineering throughput by making AI-assisted development an organizational platform rather than an individual autocomplete habit. The talk’s key components are: executive pressure, one main agent platform, internal skills/plugins, hooks, telemetry, backtesting, automatic review for safe PRs, and a culture where engineers “move up the stack” from typing code to designing/operating agent workflows.

The thesis is strong as an operating model. The weak point is measurement: “code changes per R&D person” and PR throughput can be manipulated and do not by themselves prove business value or code quality.

Comment insights

The comments are mixed and useful:

  • A top critical comment says asking people to double PRs will produce smaller PRs, not necessarily more outcome. This is a valid measurement caveat.
  • Another commenter says Intercom may not be an “outcomes over outputs” culture. This is also a fair risk if leadership overweights PR count.
  • A positive commenter from Bitloops agrees with durable, testable skills but argues context should be auto-generated and queryable, not hand-maintained markdown. That is a meaningful alternative architecture: graph/queryable context instead of manually curated skill files.
  • Several comments appear unrelated/spammy book promotions and should not be treated as audience signal.

Practical synthesis: adopt the platform/flywheel, but pair it with outcome metrics and context freshness mechanisms.

Deep research

External sources support several parts of the talk:

  • Intercom’s older “Shipping is your company’s heartbeat” post argues frequent shipping creates faster customer feedback and that deployment cost has approached zero. That source supports Scanlan’s claim that Intercom has long valued shipping cadence, not just post-LLM velocity. Source: Intercom, “Shipping is your company’s heartbeat,” https://www.intercom.com/blog/shipping-is-your-companys-heartbeat/
  • Anthropic’s Claude Code documentation describes Claude Code as a coding assistant that can work across files/tools, supports CLI/IDE/web surfaces, and integrates with CI/CD, channels, Chrome, GitHub Actions, and Agent SDK. This supports the feasibility of treating Claude Code as a platform surface. Source: https://docs.anthropic.com/en/docs/claude-code/overview
  • Claude Code Skills documentation says skills are markdown instructions that load when relevant and can contain procedures, examples, scripts, and validation. This supports the talk’s emphasis on durable reusable skills. Source: https://docs.anthropic.com/en/docs/claude-code/skills
  • Claude Code Hooks documentation confirms lifecycle events such as PreToolUse, PostToolUse, SessionStart, Stop, and hooks that can block destructive commands. This supports the “hooks/guardrails” portion. Source: https://docs.anthropic.com/en/docs/claude-code/hooks
  • Claude Code security docs confirm permission-based architecture, read-only defaults, sandboxing, write restrictions, audit considerations, and MCP security caveats. This supports the need for controls, but does not prove Intercom’s compliance claims. Source: https://docs.anthropic.com/en/docs/claude-code/security
  • OpenTelemetry docs define observability as understanding system state through traces, metrics, and logs. This supports using production-style telemetry on agent workflows. Source: https://opentelemetry.io/docs/what-is-opentelemetry/

Contradicting/limiting evidence:

  • The public docs show capabilities and controls, but they do not independently verify Intercom’s claimed 2x throughput, 17.6% auto-approval rate, code quality increase, or compliance outcomes.
  • The comments raise a classic Goodhart’s Law risk: PR count can rise while value does not. The transcript itself acknowledges “every measure is bad” once measured.

Verdict

Claim: Intercom doubled PR/code-change throughput through AI platformization.

Verdict: Mixed / plausible but not independently proven.
Confidence: Medium.
Supporting evidence: the talk shows internal dashboards and concrete platform artifacts; external docs support the feasibility of skills/hooks/Claude Code workflows.
What is overclaimed: public evidence does not let us verify the internal metric, causality, or whether customer outcomes doubled.
What is underclaimed: the organizational system — staffing a 2x platform team, telemetry, backtesting, and skill maintenance — is likely more important than the specific model.

Claim: Choose one main agent platform instead of spreading effort across many tools.

Verdict: Agree with caveats.
Confidence: Medium-high.
Practical takeaway: standardize enough to compound internal skills, hooks, and support; keep artifacts portable to avoid vendor lock-in.

Claim: Agents should be connected to everything an engineer can access.

Verdict: Mixed.
Confidence: Medium.
Agree: agents need real tools and context to perform senior engineering work.
Caution: access should be least-privilege, logged, scoped, and staged. Anthropic’s own security docs emphasize permission controls and MCP trust verification.

Claim: Automatic code approval can be compliant and reduce risk.

Verdict: Mixed / possible but high bar.
Confidence: Medium-low externally.
Practical takeaway: start with low-risk PR categories, deterministic tests, path restrictions, audit logs, and human sampling. Do not generalize to all code review.

Screen-level insights

  • 0:38 growth chart: The slide compares Intercom’s growth recovery against broader SaaS decline, with “CEO change” and “Fin launch” markers. This visual matters because the talk frames AI adoption as tied to business recovery, not only developer tooling.
  • 3:40–4:11 “2x” slides: The large “2x” visual anchors the explicit target: double engineering throughput within a year. The simplicity is rhetorically powerful but also hides measurement complexity.
  • 12:53 “Run Less (AI) Software”: The slide says Intercom deprecates internal tools when vendor replacements become first-class. This connects to the transcript’s platform conservatism: build durable Intercom-specific skills, not a sprawling custom agent framework.
  • 19:27 plugin marketplace slide: The visible numbers — 42 plugins, 312 unique skills / 440 total, 47 MCP integrations, 751 eval test files, 79 hooks, 215 contributors, 1,674 commits, 119 recent committers — are the strongest screen evidence for platform scale. The visual step matters because these numbers are not all spoken clearly in the transcript and reveal the magnitude of internal investment.

Visible tools/UI: Claude Code, internal Claude Code plugins, MCP integrations, Honeycomb telemetry references, S3 transcript mining, Codex for code review, Rails/React conventions, internal skill marketplace.

Verification notes

Verification passes performed:

  1. Source/evidence audit: Cross-checked claims against Intercom’s shipping post, Anthropic Claude Code overview/skills/hooks/security docs, OpenTelemetry docs, and GitHub Claude Code Action. Evidence supports feasibility and operating pattern, but not Intercom’s private metrics.
  2. Transcript/comment/frame fidelity audit: Checked transcript sections around 2x target, platform choice, skills/hooks, auto-review, telemetry, and comments. Key frame observations are tied to visible slides and nearby transcript.
  3. Hallucination/overclaim audit: Marked private Intercom metrics as plausible but unverified. Avoided claiming public proof of 2x, compliance, or code-quality improvement.
  4. Actionable Insights audit: The top section includes concrete setup steps, links, checklists, tools, evaluation criteria, and cautions. It is workflow-ready rather than a summary.

Residual uncertainty: internal dashboards and compliance claims are not externally auditable from available public sources; comment set is small and includes spam.

  • Actionable Insights audit: expanded to the newer detailed format with fuller implementation notes, evaluation checks, and cautions where the existing evidence supports elaboration.