Source: Android Developers Blog (Google) — 2026-07-08

Summary

Google rebuilt its Android-coding LLM leaderboard on the new open-source "Harbor" evaluation framework, replacing its earlier mini-swe-agent-v1 harness, and added eight new models to the comparison — including Claude Fable 5, Claude Sonnet 5, Claude Opus 4.8, GLM 5.2, Kimi K2.7 Code, MiniMax M3, and Qwen 3.7 Plus/Max. Claude Fable 5 currently leads the updated leaderboard at 84.5% accuracy on Android-specific coding tasks.

Key Takeaways

• Google switched its Android-coding benchmark harness from a proprietary mini-swe-agent-v1 setup to the open-source Harbor framework — making the eval itself independently reproducible.
• Eight models were newly added to the leaderboard in this update, spanning Anthropic, Zhipu (GLM), Moonshot (Kimi), MiniMax, and Alibaba (Qwen).
• Claude Fable 5 tops the refreshed leaderboard at 84.5% accuracy on Android-specific coding tasks.
• A harness swap alone can shift rankings meaningfully — Google is explicit that this is a re-baseline, not a like-for-like continuation of prior scores.
• Signals growing standardization pressure toward open, swappable eval harnesses (Harbor here, similar to SWE-bench-style efforts elsewhere) rather than vendor-controlled leaderboards.

Reel Script

Hook Google just changed the ruler it uses to measure AI coding models — and the new leaderboard has a clear leader. If you've been comparing model benchmark scores across different companies, here's why that comparison is shakier than it looks.

Core Concept An "eval harness" is the scaffolding that actually runs a model against a coding task — it gives the model tools, decides when it's done, and grades the result. Google swapped its harness from something proprietary, called mini-swe-agent-v1, to an open one called Harbor. It's like switching from a house-brand bathroom scale to a certified one — the same person can read a different number depending on which scale they step on, so you can't trust a leaderboard's absolute numbers until you know what scale produced them. That's why Google explicitly flagged this as a re-baseline, not a continuation of old scores — every model on the new board got re-run under identical, published conditions.

Hands-On The concrete result: eight new models added, spanning Claude Fable 5, Claude Sonnet 5, Claude Opus 4.8, GLM 5.2, Kimi K2.7 Code, MiniMax M3, and Qwen 3.7 Plus/Max — and Claude Fable 5 comes out on top at 84.5% accuracy on Android-specific coding tasks under the new Harbor harness. Because Harbor is open-source, anyone can pull the same harness and re-run it themselves rather than trusting a vendor's self-reported number — that reproducibility is arguably the bigger story than any single model's score, since it's what lets the 84.5% figure actually mean something comparable across labs.

Takeaway My verdict: the specific number matters less than the fact you can now verify it yourself. If a benchmark claim doesn't come with an open, swappable harness like this, treat the score as marketing until proven otherwise. Follow for more on how to actually verify AI capability claims instead of taking them on faith.

Discussion

(No questions yet — ask follow-ups via a Claude Code chat session on this repo; answers get appended here.)