Source: Elastic Security Labs (Dhrumil Patel) — 2026-08-04
Summary
Elastic Security Labs built an internal evaluation framework that runs identical prompts against every candidate LLM inside a live Elastic Security deployment rather than a synthetic sandbox detached from the real product, capturing full tool-call traces, latency, and token counts before blind-judging the outputs. The framework tests three real SOC workflows — Agent Builder, Attack Discovery, and automatic SIEM-rule migration between formats like Splunk, QRadar, and Sentinel into Elastic's own rule syntax — using a repeatable synthetic intrusion scenario called "Chrysalis," which includes an EICAR-hash malicious loader, the industry-standard harmless test file used to verify detection systems without deploying real malware. Because every model faces the identical seeded incident inside the actual product surface, Elastic can compare models on how they actually perform as SOC agents rather than on abstract benchmark tasks.
Key Takeaways
- The eval runs inside a live Elastic Security deployment, not an isolated synthetic sandbox — models are tested against the real product surface they'd actually operate in.
- Every candidate LLM receives identical prompts, and the framework captures full tool-call traces, latency, and token counts for each run, giving Elastic a like-for-like comparison beyond just final-answer accuracy.
- Model outputs are blind-judged, meaning the judge doesn't know which model produced which output, reducing bias toward a particular vendor or brand.
- Three real SOC workflows are evaluated: Agent Builder (constructing detection/response agents), Attack Discovery (finding attacks in telemetry), and automatic SIEM-rule migration (translating detection rules from Splunk, QRadar, or Sentinel into Elastic's own rule format).
- The seed data is a synthetic scenario named "Chrysalis," built to be repeatable so every model under test faces the exact same incident rather than random or varying data.
- Chrysalis includes an EICAR-hash malicious loader — EICAR is the standard industry test file used to verify antivirus and detection tooling works without using or deploying actual malware.
Reel Script
Hook: Most AI security benchmarks test a model on a quiz. Elastic tested theirs on the job — dropping every candidate LLM into a real, running security deployment and making it actually investigate an incident, not answer questions about one.
Core Concept: The standard way vendors benchmark an LLM for security work is to hand it a synthetic dataset — a curated set of questions or logs, detached from any real product, and score whether the answer matches an expected output. Elastic's argument is that this misses the thing that actually matters for a SOC agent: how it behaves when it has to call real tools, inside a real product, under time pressure. So their eval framework runs every candidate model inside an actual live Elastic Security deployment, not a mockup, and gives every model the identical prompt against three genuine workflows a security analyst does: Agent Builder, where the model has to construct a working detection or response agent; Attack Discovery, where it has to find an actual attack pattern in telemetry; and SIEM-rule migration, where it has to translate a detection rule written for a competitor's syntax — Splunk, QRadar, Sentinel — into Elastic's own rule format without breaking the rule's logic. Instead of just grading the final answer, the framework records the model's full tool-call trace — every action it took to get there — plus latency and token counts, then a blind judge scores the output without knowing which model produced it.
Hands-On: The part worth diagramming is the seed data and the pipeline shape. Elastic built a repeatable synthetic intrusion called "Chrysalis" — a fabricated but realistic attack scenario that includes an EICAR-hash malicious loader. EICAR is the file security vendors have used for decades to test that antivirus and detection tools actually fire, without anyone needing to handle real malware — it's a known, harmless string that every scanner is built to flag as if it were dangerous. Using Chrysalis as the seed means every model under test walks into the exact same incident, which is what makes the comparison fair: model A and model B aren't facing different levels of difficulty by chance. The pipeline itself is a straight line worth sketching: identical prompt goes into the live SOC deployment, the model executes its investigation using real tools inside that deployment, every tool call plus latency and token count gets logged as a trace, and that trace plus the final output goes to a blind judge who scores it without seeing the model's identity.
Takeaway: A benchmark that only measures whether a model gives the right final answer misses whether it got there efficiently, safely, and using the tools a real analyst would trust — Elastic's live-deployment method is a meaningfully harder and more honest test than a synthetic quiz. If you're evaluating LLMs for security tooling, tool-call traces inside a real deployment should be part of your bar, not just accuracy scores.