Source: arXiv — 2026-08-03
Summary
Most existing vulnerability-detection benchmarks test models on isolated code snippets rather than realistic, full-repository exploration. VulnGym instead requires a coding agent to autonomously navigate an entire repository, build context, and locate and provide evidence for real vulnerabilities — rather than simply classifying a pre-isolated snippet. The benchmark is built from 184 reviewed real-world security advisories, spanning 408 total vulnerability entries across 23 open-source repositories.
Key Takeaways
- VulnGym evaluates coding agents on repository-level vulnerability detection: the agent must explore files, build context, and locate real vulnerabilities on its own, rather than being handed an already-isolated vulnerable snippet.
- It requires the agent to produce evidence for what it finds, not just a classification label, closer to how a human security reviewer would work.
- The benchmark is built from 184 manually reviewed real-world security advisories, distilled into 408 total vulnerability entries across 23 open-source repositories.
- Snippet-level benchmarks can overstate a model's real-world usefulness because they remove the hardest part of the task — finding where in a large, unfamiliar codebase the vulnerability actually lives — which is exactly the gap VulnGym is designed to expose.