Github

Leaderboard

We select harnesses and inference endpoints to bring out each model’s best performance.

Overall results

Ranked by task pass rate, with mean cost and median runtime reported per task. A task passes only when all its cases pass.

ModelHarnessPass rateTokensCostTimeCachePer-task result
1
OpenAI
GPT-6 Astra
Codex
Codex
34.1%
15/44
2.3M$3.4811m 48s97%
2
Claude
Claude Opus 5.5
Claude Code
Claude Code
27.3%
12/44
10.9M$3.9720m 50s98%
3
DeepSeek
DeepSeek Flash
open
DSH
15.9%
7/44
29.0M$0.2323m 56s99%
4
Z.ai
GLM-5.3 Flash
open
ZCode
9.1%
4/44
24.8M$0.821h 02m99%
5
Qwen
Qwen3.8 Max
Codex
Codex
6.8%
3/44
16.6M$5.121h 04m98%
6
MoonshotAI
Kimi K3
open
Kimi
Kimi Code
4.5%
2/44
18.1M$7.921h 01m98%
1
OpenAI
GPT-6 Astra
Codex
Codex
34.1%
15/44
Tokens
2.3M
Cost
$3.48
Time
11m 48s
Cache
97%
2
Claude
Claude Opus 5.5
Claude Code
Claude Code
27.3%
12/44
Tokens
10.9M
Cost
$3.97
Time
20m 50s
Cache
98%
3
DeepSeek
DeepSeek Flash
open
DSH
15.9%
7/44
Tokens
29.0M
Cost
$0.23
Time
23m 56s
Cache
99%
4
Z.ai
GLM-5.3 Flash
open
ZCode
9.1%
4/44
Tokens
24.8M
Cost
$0.82
Time
1h 02m
Cache
99%
5
Qwen
Qwen3.8 Max
Codex
Codex
6.8%
3/44
Tokens
16.6M
Cost
$5.12
Time
1h 04m
Cache
98%
6
MoonshotAI
Kimi K3
open
Kimi
Kimi Code
4.5%
2/44
Tokens
18.1M
Cost
$7.92
Time
1h 01m
Cache
98%

Pass rate, cost, and time

Compare task pass rates with mean cost and median runtime across all tasks, including failures.

OpenAI
GPT-6 Astra
Claude
Claude Opus 5.5
DeepSeek
DeepSeek Flash
Z.ai
GLM-5.3 Flash
Qwen
Qwen3.8 Max
MoonshotAI
Kimi K3
Cost includes subagents and is averaged over all 44 tasks. Time is the median per task run.

Results by application

Passed tasks out of the total for each application.

Actual Budget
Afilmory
Cal.diy
Dify
Discourse
Documenso
Dolibarr
FreshRSS
GLPI
Gitea
Grist
Hoppscotch
Kimai
LimeSurvey
LobeHub
Metabase
Navidrome
NetBox
OpenEMR
Oppia
Penpot
Plane
Saleor Dashboard
Snipe-IT
ownCloud Infinite Scale
OpenAI
GPT-6 Astra
0/1
1/2
0/1
0/1
0/2
1/1
1/5
0/1
1/2
1/4
1/1
1/1
0/1
0/1
0/1
0/3
0/1
1/2
1/1
1/2
1/3
1/3
2/2
1/1
0/1
Claude
Claude Opus 5.5
0/1
1/2
0/1
0/1
0/2
0/1
1/5
0/1
1/2
2/4
0/1
0/1
1/1
0/1
0/1
0/3
1/1
1/2
1/1
1/2
1/3
1/3
0/2
0/1
0/1
DeepSeek
DeepSeek Flash
0/1
1/2
0/1
0/1
0/2
0/1
0/5
0/1
0/2
0/4
1/1
0/1
1/1
0/1
0/1
0/3
1/1
0/2
0/1
0/2
1/3
0/3
1/2
1/1
0/1
Z.ai
GLM-5.3 Flash
0/1
0/2
0/1
0/1
0/2
0/1
0/5
0/1
0/2
1/4
0/1
0/1
1/1
0/1
1/1
0/3
0/1
0/2
0/1
0/2
0/3
0/3
1/2
0/1
0/1
Qwen
Qwen3.8 Max
0/1
0/2
0/1
0/1
0/2
0/1
1/5
0/1
1/2
0/4
0/1
0/1
0/1
0/1
0/1
0/3
0/1
0/2
0/1
0/2
0/3
0/3
1/2
0/1
0/1
MoonshotAI
Kimi K3
0/1
1/2
0/1
0/1
0/2
0/1
0/5
0/1
0/2
0/4
0/1
0/1
0/1
0/1
0/1
0/3
0/1
0/2
0/1
0/2
0/3
0/3
1/2
0/1
0/1
OpenAI
GPT-6 Astra
Actual Budget0/1
Afilmory1/2
Cal.diy0/1
Dify0/1
Discourse0/2
Documenso1/1
Dolibarr1/5
FreshRSS0/1
GLPI1/2
Gitea1/4
Grist1/1
Hoppscotch1/1
Kimai0/1
LimeSurvey0/1
LobeHub0/1
Metabase0/3
Navidrome0/1
NetBox1/2
OpenEMR1/1
Oppia1/2
Penpot1/3
Plane1/3
Saleor Dashboard2/2
Snipe-IT1/1
ownCloud Infinite Scale0/1
Claude
Claude Opus 5.5
Actual Budget0/1
Afilmory1/2
Cal.diy0/1
Dify0/1
Discourse0/2
Documenso0/1
Dolibarr1/5
FreshRSS0/1
GLPI1/2
Gitea2/4
Grist0/1
Hoppscotch0/1
Kimai1/1
LimeSurvey0/1
LobeHub0/1
Metabase0/3
Navidrome1/1
NetBox1/2
OpenEMR1/1
Oppia1/2
Penpot1/3
Plane1/3
Saleor Dashboard0/2
Snipe-IT0/1
ownCloud Infinite Scale0/1
DeepSeek
DeepSeek Flash
Actual Budget0/1
Afilmory1/2
Cal.diy0/1
Dify0/1
Discourse0/2
Documenso0/1
Dolibarr0/5
FreshRSS0/1
GLPI0/2
Gitea0/4
Grist1/1
Hoppscotch0/1
Kimai1/1
LimeSurvey0/1
LobeHub0/1
Metabase0/3
Navidrome1/1
NetBox0/2
OpenEMR0/1
Oppia0/2
Penpot1/3
Plane0/3
Saleor Dashboard1/2
Snipe-IT1/1
ownCloud Infinite Scale0/1
Z.ai
GLM-5.3 Flash
Actual Budget0/1
Afilmory0/2
Cal.diy0/1
Dify0/1
Discourse0/2
Documenso0/1
Dolibarr0/5
FreshRSS0/1
GLPI0/2
Gitea1/4
Grist0/1
Hoppscotch0/1
Kimai1/1
LimeSurvey0/1
LobeHub1/1
Metabase0/3
Navidrome0/1
NetBox0/2
OpenEMR0/1
Oppia0/2
Penpot0/3
Plane0/3
Saleor Dashboard1/2
Snipe-IT0/1
ownCloud Infinite Scale0/1
Qwen
Qwen3.8 Max
Actual Budget0/1
Afilmory0/2
Cal.diy0/1
Dify0/1
Discourse0/2
Documenso0/1
Dolibarr1/5
FreshRSS0/1
GLPI1/2
Gitea0/4
Grist0/1
Hoppscotch0/1
Kimai0/1
LimeSurvey0/1
LobeHub0/1
Metabase0/3
Navidrome0/1
NetBox0/2
OpenEMR0/1
Oppia0/2
Penpot0/3
Plane0/3
Saleor Dashboard1/2
Snipe-IT0/1
ownCloud Infinite Scale0/1
MoonshotAI
Kimi K3
Actual Budget0/1
Afilmory1/2
Cal.diy0/1
Dify0/1
Discourse0/2
Documenso0/1
Dolibarr0/5
FreshRSS0/1
GLPI0/2
Gitea0/4
Grist0/1
Hoppscotch0/1
Kimai0/1
LimeSurvey0/1
LobeHub0/1
Metabase0/3
Navidrome0/1
NetBox0/2
OpenEMR0/1
Oppia0/2
Penpot0/3
Plane0/3
Saleor Dashboard1/2
Snipe-IT0/1
ownCloud Infinite Scale0/1