Source: Databricks Blog — 2026-07-08
Summary
Databricks built a proprietary coding-agent benchmark from real merged pull requests against its own multi-million-line internal codebase (Python, Go, TypeScript, Scala), deliberately avoiding public leaderboards like SWE-Bench, to measure real-world cost versus quality. The results were substantial enough that Databricks switched its default coding-agent model to GLM 5.2.
Key Takeaways
- Top tier (82-90% pass rates) includes Opus 4.8, GLM 5.2, and GPT 5.5, with the Chinese open-weight GLM 5.2 statistically tying Opus 4.8 on quality at $1.28/task vs. $1.94/task, a 34% cost reduction.
- Key conclusion: "token price is a poor proxy for real-world cost" — larger models can be more token-efficient end-to-end than their sticker price suggests.
- Harness choice materially affects both cost and quality — simpler harnesses outperformed more complex ones on Databricks' internal workloads in many cases.
- Databricks acted on the findings by making GLM 5.2 its default coding-agent model, not just publishing a leaderboard.