🔍 Read the full analysis: What A GPT-6 Astra Review Added To Claude’s Model Edit Tests on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI’s GPT-6 Astra, running in Codex, independently reviewed an Anthropic Claude Opus 5.5 experiment that edited a local model’s weights. The review found reproducible problems in the test rig and challenged several claims; the published 10-task results showed more answered tasks, but the legal score remained zero.
OpenAI’s GPT-6 Astra, running in Codex, found reproducible flaws in the benchmark setup and weight-conversion process used in an Anthropic Claude Opus 5.5 experiment on local model Ornith 1.5, according to a report published by ThorstenMeyerAI.com. The review did not run a competing model-edit experiment; it examined the Claude session’s work and prompted corrections to both the test code and claims about the results.
The Claude session used a directional ablation to weaken what the experiment described as a “loop direction” in Ornith 1.5’s weights, without further training. The site says that, across 10 published Lexassay tasks, the edited model answered all 10, compared with four for the unedited model. Its legal score remained zero. The figures were updated November 9 to refer to the 10 published tasks, and the runs are publicly available on Lexassay.
GPT-6 Astra’s first review, on November 4, reported four reproducible issues. Resuming a run could reuse an answer despite a change to the model or sampling settings; an unknown task ID could yield a successful run with no evaluations; a comma conflict broke the actual command line despite passing parser unit tests; and converting weights changed values even when the edit strength was zero. The report says Claude’s session changed the command syntax, rejected unknown task IDs and stopped re-quantising edited tensors into the compact 6-bit format.
The reviewer also checked the revised work. In the second round, all 12 of the author’s tests passed, while four of the reviewer’s six provenance checks failed. GPT-6 Astra measured a separate format conversion on one matrix: it moved 0.51% of the matrix and changed 915,360 of 1,048,576 values. That measurement describes conversion, not legal performance or an error rate. The report says the reviewer also corrected wording about zero-strength edits, citation averages, the extent of its role and the causal interpretation of a control.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
Why Independent Review Changed the Results
The review matters because a benchmark can produce orderly-looking results while its run controls are wrong. If an interrupted test silently reuses answers from a different setup, or a typo completes with zero evaluations, the resulting record may not describe the experiment researchers think they ran. A green unit-test result did not catch the command-line conflict; testing the real entry point did.
The weight-conversion finding also bears on attribution. If converting a model changes its values even when the intended edit is zero, observed changes cannot automatically be assigned to the edit itself. The report says the conversion method was changed, but the measured difference is not evidence that the model became better or worse at law.
For readers evaluating the headline result, the key distinction is between answer coverage and legal accuracy. The edited model answered more of the 10 tasks, but its legal score stayed at zero. The review makes the experiment’s process more inspectable; it does not establish that the edit improved legal reasoning or that the result will generalize beyond this task set.
Popular right now
KitchenAid Artisan 5-Quart Tilt-Head Stand Mixer, Blue Velvet
Samsung 43-Inch M70H Series 4K Mini LED Smart TV (2026)
Philips 3300 Series Fully Automatic Espresso Machine with Milk Frother, Grinder, Hot and Iced Coffee Presets, SilentBrew, 15-Bar Pressure (E
As an affiliate, we earn on qualifying purchases.
How Claude and Codex Split the Work
Two AI sessions began work on the experiment on November 4. After the organizers realized they were pursuing the same project, Claude remained responsible for the plan, implementation, edit, benchmark runs and blind grading by three Claude agents. GPT-6 Astra was assigned a separate review role, described in its report as: “Review and validate the other session’s implementation.”
The reviewer did not alter the main implementation or inspect the hidden benchmark prompts, topics, answers or reference solutions, according to the report. Instead, it wrote separate checks and gave findings to the Claude session, which made the fixes. The source says preparatory files from GPT-6 Astra’s earlier work were set aside and that no second edited model resulted from them. This distinction is why the review should not be described as a parallel model-improvement experiment.
The experiment was inspired by a MIT CSAIL post on weight-space programming, according to ThorstenMeyerAI.com. The site’s account concerns a particular local model, edit and published task set; it does not establish that the method works generally. The article also states that the work is an AI experiment, not legal advice.
“Review and validate the other session’s implementation.”
— GPT-6 Astra’s report, quoted by ThorstenMeyerAI.com
What the Review Did Not Establish
The published account does not establish that the edit improved legal performance: the reported legal score remained at zero across the 10 published tasks. It also does not show whether the task results generalize to other benchmarks, models or legal uses. The source describes blind grading by three Claude agents, but supplies no independent fourth grader’s assessment.
The review had limits of its own. According to the article, GPT-6 Astra did not rerun every test during its final documentation step and measured the matrix conversion on a CPU rather than through the production GPU path. The meaning of the four failed provenance checks is not detailed in the supplied material, so they should not be treated as evidence of four additional benchmark failures. The report also says one control was insufficient to support the strength of the original causal claim.
It is not clear from the supplied account whether further review or additional experiments are planned, or whether the updated results have been independently replicated outside the published Lexassay runs. Those open questions limit what can be concluded from the reported task counts.
Further Checks Needed to Test the Claim
The immediate record is public: the source says the 10 Lexassay task runs, including the weight-edited model, an edit control and model-card sampling, can be inspected on Lexassay.com. Readers can compare the reported outcome with those runs, while keeping in mind the review’s stated limits and the difference between answering a task and earning a legal score.
The next evidentiary step would be additional runs that test whether the reported difference persists under controls for model settings and weight conversion, along with independent evaluation of legal accuracy. ThorstenMeyerAI.com’s supplied account does not announce a timetable for such work. Until further results are reported, the confirmed development is the review and the fixes it prompted—not a demonstrated general improvement in legal model performance.
Key Questions
What did GPT-6 Astra review?
It reviewed the implementation and test process for a Claude Opus 5.5-led experiment that edited the weights of local model Ornith 1.5. It did not run a competing model-improvement experiment, according to the report.
What did the model-edit experiment report?
ThorstenMeyerAI.com says the edited model answered all 10 published Lexassay tasks, compared with four for the unedited model. The legal score remained at zero, so the answer count does not show improved legal accuracy.
What problems did the review find?
The first review found four issues: resumed runs could reuse answers from a different setup, unknown task IDs could produce empty successful runs, a real command-line conflict escaped unit tests, and weight conversion altered values even with edit strength set to zero.
Did GPT-6 Astra inspect hidden benchmark tasks?
The source says the reviewer’s second review stated it did not inspect hidden prompts, topics, answers or reference solutions. It reviewed the implementation using separate checks.
Does the review prove that the edit works?
No. It documents faults, corrections and a result on 10 published tasks, but does not establish general performance gains or improved legal reasoning. The reported legal score stayed at zero, and further independent testing is not detailed.
Source: ThorstenMeyerAI.com
