🔍 Read the full analysis: FrontierCode: How To Think Like A Maintainer Reviewing AI Code on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, testing 23 models on repository tasks judged by maintainer-style criteria. Claude Opus 5.5 led with a 52.2% score and passed every blocking check in 57.7% of attempts; the results show that benchmark success does not yet make AI-generated changes reliably ready to merge.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark designed to judge whether a repository change would meet maintainer review standards, not just pass tests. Claude Opus 5.5 ranked first with a 52.2% score, but it cleared every blocking check in only 57.7% of attempts, indicating that even the leading result is not a reliable basis for hands-off merging.
The evaluation, added by Artificial Analysis on October 12, 2026, uses the benchmark’s 100-task Main set. In the results reported the following day, GPT-6.1 Sol placed second at 49.1%, followed closely by Claude Fable 5.1 at 49.0%. Nine models from six labs scored between 43% and 52%; results fell below that group, with Grok 4.7 at 33.8% and Nemotron 3 Ultra at 10.5%.
FrontierCode’s score is not the same as its pass rate. Each task is assessed across checks that include hidden tests, command checks, test quality, scope limits and an LLM code review against maintainers’ criteria. Failing any blocking check gives an attempt a score of zero; attempts that pass all blocking checks receive a score from 70% to 100%, with quality checks affecting the result. Opus 5.5’s 57.7% blocking-check pass rate means it did not clear those checks in 42.3% of attempts.
Cost varied widely. Artificial Analysis reported $0.09 per task for GPT-6 Luna (max) and $10.15 for Opus 5.5. Its cost-frontier models—those not beaten by another model that is both cheaper and higher-scoring—also included MiMo-V2.6-Pro at 45.4% for $0.18 and GPT-6.1 Sol at 49.1% for $0.81. Moving from Sol to Opus added 3.1 score points at about 12.5 times the reported cost.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Review Checks Matters
The results address a practical question for engineering teams: can an AI agent produce a change that is ready for a maintainer to accept? FrontierCode’s answer is more demanding than a conventional test pass. It penalizes changes that exceed scope, introduce weak tests or otherwise fail review criteria, even when some tests run successfully.
For teams considering AI coding tools, the gap between the top score and a complete pass rate matters. A score of 52.2% does not mean that 52.2% of all generated code is correct in every setting; it is a result on this benchmark and its particular tasks and scoring rules. Still, the 57.7% blocking-check rate shows why human review remains relevant in the tested workflow. Cost also changes the purchasing question: Opus led the results, but a team may weigh its small lead over Sol against a much higher per-task cost.
The benchmark also suggests that general model rankings may not predict repository-review performance. In the source’s comparison, Claude Sonnet 5.5 ranked second on the Artificial Analysis Intelligence Index among the listed models but fifth on FrontierCode. DeepSeek V4.1 Flash scored more strongly on FrontierCode than its Intelligence Index position might suggest. Those comparisons are specific to the models and results listed, not a universal measure of coding ability.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Repository Changes
FrontierCode was built by Cognition and open-source maintainers and announced in June 2026. Version 1.1 followed in July, with rules barring agents from looking up solutions online. The full benchmark contains 150 tasks; Artificial Analysis ran its 100-task Main set. More than 20 maintainers created tasks from 36 repositories, including Celery, Budibase, uppy and Mattermost. The source says maintainers spent more than 40 hours on each task. The tasks are private, limiting the chance that models encountered the exact solutions in training.
Artificial Analysis ran models through the same open-source agent harness, Stirrup, with a limit of 250 turns per attempt and three attempts per task. The review check used GPT-6 Luna (medium) as the judge. The benchmark also uses a trajectory scanner to flag attempts that appear to look up solutions; flagged attempts receive zero. These design choices make the score a product of the benchmark’s task selection, limits and evaluation process as well as a model’s code generation.
Cognition’s announcement described the distinction this way: “Where others grade like a CI, FrontierCode grades like a tech lead.” That is the benchmark’s framing, not evidence that its scores capture every maintainer’s preferences or every software project’s requirements.
““Where others grade like a CI, FrontierCode grades like a tech lead.””
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement
software maintainer review software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of the Current Results
The results do not establish how models perform across all repositories or real-world pull requests. They cover the 100-task Main set and use a specific harness, attempt limit and review judge. The source does not provide enough information here to determine how much scores might change under different task mixes, reviewer criteria or agent settings.
Several models were missing from the leaderboard as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the rankings are not a comparison of every prominent model. The benchmark’s claim that nine of its 100 tasks were not solved by any model on any attempt also applies to the tested attempts, not necessarily to what other systems or future versions could achieve.
It is also unclear from the reported results how closely benchmark scores predict acceptance rates in production teams. Maintainers and projects can apply different standards, and a benchmark pass is not a guarantee that a patch is safe, maintainable or appropriate for a particular codebase.
As an affiliate, we earn on qualifying purchases.
Missing Models and Future Runs
The next point of comparison is whether the absent models receive FrontierCode results and how later model versions perform under the same evaluation. The source identifies Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview as having no result as of October 13, but does not give a schedule for their evaluation.
For engineering teams, the immediate step is to treat the leaderboard as a benchmark comparison rather than a replacement for review. Further results could clarify whether the close cluster near the top persists and whether lower-cost models remain competitive as systems change. Until then, the published figures support a limited conclusion: the leading tested model did not pass every blocking check on a substantial share of attempts, and teams should not infer that AI-generated pull requests can be merged without human judgment.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It evaluates whether an AI agent’s change meets a combination of correctness, regression, test-quality, scope and code-review checks on repository tasks. It is designed to go beyond whether a patch makes tests pass.
Which model ranked first?
Claude Opus 5.5 (max with fallback) led Artificial Analysis’s reported results with a 52.2% FrontierCode score. It passed all blocking checks in 57.7% of attempts.
Does a 52.2% score mean Opus passed 52.2% of tasks?
No. The benchmark’s score incorporates blocking and quality checks. The reported blocking-check pass rate was 57.7%; the 52.2% score is a separate benchmark result.
How much did the leading models cost per task?
Artificial Analysis reported $10.15 per task for Opus 5.5 and $0.81 for GPT-6.1 Sol. These are the source’s reported evaluation costs and may not describe the cost of every use in a different setup.
Does a FrontierCode pass mean a real pull request can be merged automatically?
No. The results concern performance on a defined benchmark and do not establish that a patch will meet every project’s standards. Human review remains relevant, and the published results do not show that automated merging is reliable across production repositories.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
