🔍 Read the full analysis: Can AI Break In And Fix The Lock? A Closer Look At Gemini 4 Carbon’s Security Scores on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Google says Gemini 4 Carbon scored 88.2% on SEC-bench Pro, 54.8% on ExploitGym and 71.0% on CWE-bench, testing vulnerability discovery, exploitation and repair. Google ran the first two evaluations; Collinear AI operates the third. Carbon is being offered first to a small group of vetted cyber defenders, and no independent public evaluation of its security performance is available.
Google said Wednesday that its Gemini 4 Carbon model scored highly across three cybersecurity tests covering vulnerability discovery, exploit development and software repair, but two of the headline results were produced in evaluations run by Google itself. The company is initially providing access to a small group of vetted cyber defenders, reflecting the dual-use risk that a model able to find and exploit software weaknesses could also help attackers.
Google reported scores of 88.2% on SEC-bench Pro, 54.8% on ExploitGym and 71.0% on CWE-bench. The tests measure different tasks, so the figures should not be read as a single overall security rating. SEC-bench Pro asks a model to trigger known bugs in code; ExploitGym tests whether it can turn a crash into a working exploit; CWE-bench checks whether a model can locate and repair flaws without breaking existing tests.
Google said it ran SEC-bench Pro and ExploitGym using the open-source evaluation setup, with safety filters turned off. Its 88.2% SEC-bench Pro result was measured on a 183-task version, averaged over three attempts and run using Google’s Antigravity agent tool. OpenAI reported 85.4% for GPT-6 Astra on the same benchmark version in its own system card, while Gemini 4 Argon scored 83.4%. The source material gives no comparable Claude result.
On CWE-bench, which Collinear AI operates, Carbon scored 71%, compared with 68% for GPT-6 Astra, Gemini 4 Argon and Grok 4.7, and 67% for Claude Opus 5.5. Collinear runs the private test set, according to the source material, and says Artificial Analysis evaluated and published results. Google’s headline ExploitGym score used a 10-hour limit; comparisons with rival results are not directly equivalent because labs used different time limits, graders and tools.
Infographic 01 · What Carbon's three scores measure
Find, exploit, fix: three tests, and only one outside referee.
Google says Gemini 4 Carbon leads all three security benchmarks it reported. Here is what each one asks of the model, and who produced Carbon's number.
1Find · SEC-bench Pro
- The model gets
- Source code, the area to search, the bug type and the expected error. Not the input that triggers the bug.
- It counts when
- The model writes an input that triggers a known, already-fixed bug in a browser JavaScript engine.
- Carbon’s score came from
- Google, 183-task May 2026 version, average of 3 attempts Google's own run
2Exploit · ExploitGym
- The model gets
- An input that crashes the program. Standard protections switched off.
- It counts when
- The model turns the crash into its own code running on the target and reads a hidden flag; an AI judge checks it used the intended bug.
- Carbon’s score came from
- Google, one attempt per task, 10-hour limit (53.5% at 6 hours) Google's own run
3Fix · CWE-bench
- The model gets
- A real open-source code base and one instruction: audit it and fix what you find. No hint what or where.
- It counts when
- The attack stops working and every existing test still passes. Models never write exploits.
- Carbon’s score came from
- Collinear AI, 120 private tasks, all models run by Collinear Outside run
Read the fine print
- Google tested without safety filters (“evaluated without guardrails”); Anthropic and OpenAI report their cyber results with safeguards off too
- Rival numbers come from the rivals’ own system cards, with their own time limits and graders
- SEC-bench Pro and ExploitGym use public bugs with public fixes; CWE-bench keeps its test set private
Finding is not discovering
- SEC-bench Pro tells the model where to look and what kind of bug to expect
- That is closer to a detective with a tip than to a hunt for unknown flaws
- Real attacks also need targets, delivery and staying undetected; real fixes need review
The same skill that proves a bug is dangerous is the skill an attacker needs.Thorsten Meyer · my reading
Sources: Google, 28 Oct 2026; Google methodology note, Oct 2026; CWE-bench leaderboard, 28 Oct 2026; SEC-bench Pro paper; ExploitGym paper; OpenAI GPT-6 Astra system card, 3 Sep 2026. Lead margin and counts as in the article; margin is my arithmetic.
Powered by Thorsten Meyer AI
Security Scores Carry Dual-Use Risks
The results matter because the same capabilities could assist defenders finding and repairing weaknesses or help an attacker develop an exploit. Google’s decision to begin with vetted cyber defenders limits who can use the model while the company evaluates its security implications. The source says Google also participates in voluntary U.S. government pre-release testing.
The benchmarks offer evidence of performance on defined tasks, not proof that Carbon can independently secure a company’s systems or outperform human experts in live environments. For organizations, the repair score still leaves 29% of CWE-bench tasks unsuccessful on the first try, according to the source’s account. AI-generated fixes would need review and testing before deployment. For ordinary users, the report describes no immediate change to consumer security; keeping software updated remains a practical protection.
The offensive score warrants particular care in interpretation. Google counted Carbon turning 53.5% of known bugs into working exploits within six hours, while its 54.8% headline used a 10-hour limit. Anthropic reported 34.5% for Claude Opus 5.5 at six hours, but that result came from Anthropic’s own report. With different evaluation conditions and no public independent test of Carbon, the figures do not settle how the models compare in real-world security work.
Popular right now
De'Longhi Rivelia Fully Automatic Espresso Machine with Milk Frother, White
Vitamix 5200 Professional-Grade Blender, 64 oz. Container, Black
Ninja Specialty Coffee Maker CM401, 10-Cup Carafe, Fold-Away Frother, Black
As an affiliate, we earn on qualifying purchases.
Three Tests Measure Different Tasks
SEC-bench Pro was built by researchers at the University of Illinois Urbana-Champaign from previously fixed bugs in Chrome’s V8 and Firefox’s SpiderMonkey JavaScript engines, with the newest version also covering the Linux kernel. The model receives source code, a search area, a bug type and an expected error, but not the input that triggers the flaw. It must produce that input. This measures whether it can reproduce a known weakness under guided conditions, rather than find an unknown vulnerability from scratch.
ExploitGym was developed by researchers led by UC Berkeley, with contributors from the Max Planck Institute for Security and Privacy and several AI labs. Its 869 cases draw on real vulnerabilities in common programs, V8 and the Linux kernel. A model starts with a crashing input and must make it execute code and expose a hidden flag. Google’s test disabled standard protections including memory randomization, a condition that shapes what the result says about real systems.
CWE-bench takes a different approach: the model is given an open-source codebase and asked to audit and repair it, without being told where or how many flaws to expect. A fix counts only if it blocks the attack and existing tests still pass. Its 120 tasks are private, according to the source, to reduce the chance that benchmark material appears in model training data. The tests therefore span distinct stages of security work, not interchangeable measures of one skill.
“self computed”
— Google, in its methodology note
Independent Results Are Still Missing
No public independent evaluation of Gemini 4 Carbon’s security performance is described in the source material. Google ran two of the three cited tests, while the third is administered by Collinear AI; the source also says Artificial Analysis evaluated and published the CWE-bench results. The degree to which those arrangements produce results comparable across labs remains unclear.
Comparisons also have limits: rival figures are drawn from labs’ own reports, and the ExploitGym time limits, graders and tools differ. The source does not establish how Carbon would perform with standard protections enabled, on systems outside the benchmarks, or in the hands of users beyond Google’s initial vetted group. It also gives no detailed account of the access terms or the outcome of the U.S. government’s pre-release testing.
Access and Testing Will Shape the Picture
Google says it is giving Carbon to a small, vetted group of cyber defenders first and has included the model in voluntary U.S. government pre-release testing. The source does not provide a date for broader access or report findings from that government testing. Further public results, particularly evaluations conducted independently under comparable conditions, would help clarify how much the benchmark scores say about practical security work.
Until then, companies considering AI-assisted security work should treat vendor benchmark scores as bounded test results, not guarantees. They will need to review generated patches and confirm that fixes work within their own systems. There is no reported consumer product change tied to the announcement; for users, the source says routine software updates remain the immediate security measure.
Key Questions
What did Gemini 4 Carbon score?
Google reported 88.2% on SEC-bench Pro, 54.8% on ExploitGym and 71.0% on CWE-bench. The tests measure vulnerability triggering, exploit development and flaw repair, respectively.
Were the benchmark scores independently verified?
Not all of them. Google ran the SEC-bench Pro and ExploitGym evaluations. Collinear AI operates CWE-bench, and the source says Artificial Analysis evaluated and published those results. It reports no public independent test of Carbon’s security skills.
Can the model be used by the public now?
The source says Google is initially providing it to a small group of vetted cyber defenders. It does not give a date for general access or describe public availability.
Does the exploit score show Carbon can attack real systems?
It shows performance on a defined benchmark using known vulnerabilities and specified test conditions. Google’s evaluation disabled protections including memory randomization, so the score alone does not establish performance against real-world systems with normal safeguards.
What should companies take from the repair score?
The CWE-bench result indicates performance on a private set of repair tasks, not a guarantee that generated fixes are safe in production. The source says even the best model failed roughly three in ten tasks on its first attempt, supporting human review and testing of AI-written patches.
Source: ThorstenMeyerAI.com
