Google updates Android Bench with new LLMs, but Gemini still lags behind

Google’s updated leaderboard shows Gemini slipping to fifth place.

Credit:
Google

Google’s updated leaderboard shows Gemini slipping to fifth place.

Credit:

Google

However, Fable 5 and GPT 5.5 also have extremely high operating costs, chewing through more than $130 in tokens for the 100-problem, 10-run benchmark. Gemini 3.1 Pro didn’t score as high, but it only costs $87 to run the test. Gemini 3.5 Flash, which is supposed to be cheaper to run than other models, has the highest cost on the leaderboard because it took so much longer to complete the benchmark: $165 per run and a 28-hour runtime.

The Android coding performance gap for Google’s models is a problem as the company shifts many of its projects toward agentic development. Obviously, Google would prefer that Android developers use Google’s tools in their workflows, which may be why Google has reportedly been offering to buy application source code from developers for AI training.

Community collaboration

Android Bench is supposed to evolve over time, adopting new workflows to test models. Google hopes that developers will want to contribute to Android Bench by sharing benchmarks and development tasks. To make that more feasible, Google is switching to the Harbor framework. According to the company, this testing sandbox makes it easy for developers to run, evaluate, and share results for Android Bench.

Google re-ran all its previous tests with Harbor to get a new baseline for LLM performance. So there’s been some shift in the previously reported scores even though the…

Source