Source: Claude by Anthropic (Datadog case study) — 2026-07-21
Summary
Datadog's long-running Claude Code sessions — some spanning days — kept running into the same problem: an agent that needs a new tool mid-task would just write ad hoc glue code to do the job, which is fast but leaves behind unverified, one-off scripts nobody can trust for the next run. Temper is Datadog's answer: a small, verified runtime "kernel" that agents write deterministic specifications against, rather than free-form scripts, so the tools an agent builds for itself are checked before they're trusted. The project is the fourth step in a lineage — Courier, BitsEvolve, and Helix — where each earlier system pushed more of the correctness burden from human code review onto automated verification, with Helix reportedly using simulation-based verification as its primary correctness check instead of review, and hitting about 93% of peak disk throughput.
Key Takeaways
- Temper isn't the sandbox the agent runs in — it's a verified runtime specifically for the tools and glue code the agent builds while it works, addressing what happens after an agent decides "I need a new tool" mid-session.
- Every spec written for Temper passes four separate verification layers: symbolic verification, model checking, simulation, and property testing — a stacked pipeline rather than a single pass/fail check.
- The project has a real lineage inside Datadog: Courier (a distributed queuing system built entirely by hand in 2024) came first, then BitsEvolve (which tracks an evolving codebase's state and prompts Datadog's Bits AI Dev Agent to make changes, benchmarked in a feedback loop), then Helix (a Kafka-compatible streaming engine where the verification harness itself replaced code review as the primary correctness gate), then Temper as the generalized version of that pattern.
- Helix's simulation-based harness sustained millions of deterministic simulation runs and reportedly reached about 93% of peak disk throughput — a concrete signal that verification-first development can match hand-tuned system performance.
- Claude Code reportedly drives at least two-thirds of Datadog's AI coding tool usage internally, which is the volume that made ad hoc, unverified tool-building a real operational risk rather than a theoretical one.
- The underlying bet: as agent sessions get longer and more autonomous, the safety-critical question shifts from "is the agent's final output correct" to "can we verify the tools the agent built for itself along the way" — a harness-engineering problem, not a prompting problem.
Reel Script
Hook (18s, ~40 words): Datadog let Claude Code run sessions spanning multiple days — and the real risk wasn't the agent's final answer, it was the throwaway tools it built for itself along the way. Their fix is a small verified runtime called Temper.
Core Concept (75s, ~170 words): When an agent works on something for hours or days instead of minutes, it inevitably hits a point where it needs a tool that doesn't exist yet — a script to parse some format, a helper to hit an internal API. The obvious move is to let the agent just write that glue code on the spot. The problem is that code is unverified and disposable — nobody reviewed it, nobody proved it does what it claims, and it silently becomes part of the agent's trusted toolchain anyway. Datadog's answer is to give the agent a small, fixed "kernel" — Temper — and have it write formal specifications against that kernel instead of arbitrary scripts. Think of it like the difference between a mechanic improvising a fix with whatever's lying around versus using a certified machine tool that only accepts precisely specified inputs: the machine tool constrains what's possible, so what comes out is trustworthy by construction, not by hope.
Hands-On (70s, ~155 words): The concrete pipeline: every spec an agent writes for Temper has to clear four separate verification layers before it's trusted — symbolic verification, model checking, simulation, and property testing. That's not one test suite, it's four different classes of proof stacked on top of each other. Temper is actually the fourth system in a chain Datadog built up to this point. First came Courier, a distributed queuing system built entirely by hand over a year, with no AI involvement — the baseline. Then BitsEvolve, which tracks a system's evolving state and has Datadog's AI dev agent propose changes, benchmarked in a loop rather than reviewed by eye. Then Helix, a Kafka-compatible streaming engine, where the verification harness got strong enough to fully replace human code review as the primary correctness check — running millions of simulated scenarios and still landing around 93% of peak disk throughput. Temper generalizes that pattern into a reusable kernel instead of a one-off system.
Takeaway (25s, ~55 words): If your agents are running long, multi-day sessions and improvising their own tooling, that improvised code is your actual attack surface, not the agent's final output. The ROI case here is real — Datadog matched hand-tuned performance with a verified pipeline. Look at whether your harness lets agents build unverified tools by default.