🔍 Read the full analysis: Did Weight-Space AI Editing Succeed At Home? Legal Score Was Zero on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A home experiment edited the weights of the local model Ornith 1.5 35B-A3B to reduce looping. On the 10 published Lexassay tasks, it answered all 10, compared with 4 on the same setup before editing, but its legal score remained zero and its answers included 55 wrong or invented citations. The test was limited, and the results do not show that the edit improved legal knowledge.
A home experiment reported by ThorstenMeyerAI.com found that a weight edit stopped a local AI model from failing to finish many legal benchmark tasks, but its legal score remained zero. Ornith 1.5 35B-A3B answered all 10 published Lexassay tasks after the edit, up from 4 on the same setup before it; the edited answers nevertheless contained 55 wrong or invented citations.
The experiment used directional ablation: researchers recorded the model’s internal signals while it ran, identified a direction associated with looping, then removed that direction at half strength from the output weights of 27 of its 40 layers. According to the report, the process involved no additional training, new legal data, or AI reading the weights as text. It was inspired by a MIT CSAIL report on “weight-space programming,” but did not reproduce that method.
On the 10 published tasks, the edited model completed all 10, compared with 4 in the reported same-setup comparison. Its legal score stayed at 0. The report says the edited answers met 7 of 50 checked criteria, up from 3, and included 55 citations that were wrong or invented. The benchmark checks cited statutes and court decisions against primary sources. The article says all answers, grades, citation checks and experiment details are public on Lexassay.
The report also tested the unchanged model using sampling settings recommended by its model card. Under those settings, it answered 9 of 10 tasks without an edit on its original setup. The author cautions that this is not a directly comparable result: the original and edited runs used two local setups, and the unchanged model answered 8 of 10 in a published run but only 4 of 10 on the setup used for the edit comparison.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
Finishing Answers Is Not Legal Accuracy
The result illustrates a distinction that matters when people evaluate AI systems: producing a complete answer does not mean producing a correct one. A change that reduces looping may make a model more usable by allowing it to finish, but the reported legal score and citation checks show no demonstrated improvement in legal reliability.
That distinction is especially relevant in legal work, where a fabricated or incorrect citation can undermine an otherwise plausible answer. Lexassay checks claims against primary legal sources, rather than treating fluent or complete responses as evidence of competence. The experiment therefore supports only a narrow conclusion: the edit helped task completion in one test, while the available results do not show that it supplied knowledge the model lacked.
The report also highlights how evaluation settings can affect apparent performance. The unchanged model’s answer count differed across setups and sampling settings, so the headline comparison should not be read as proof that the edit alone caused every difference. One run per setting leaves substantial uncertainty about repeatability.
Popular right now
Ninja CREAMi 7-in-1 Ice Cream Maker, NC299AMZ
Ninja Crispi Microwave & Air Fryer with FusionCrisp Technology, 14-in-1, Black (MW101BK1)
Vitamix VX1 Professional-Grade Blender, 64 oz, Black
As an affiliate, we earn on qualifying purchases.
From MIT’s Weight-Space Idea to Lexassay
The test followed a report from four MIT CSAIL researchers: Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab. Their work described a method in which a general-purpose language model directly examines and edits another model’s weights. In a Qwen example, the researchers linked looping to a direction in the model’s internal signals and reported removing it; the model’s maths score rose from 7.9% to 48.1%. That was a different procedure from the home experiment, which used a more established technique based on running the model and measuring its signals.
Lexassay covers German, U.S. and EU law and is designed to test answers without internet access or tools. The benchmark has 15 questions, but the report’s figures here refer only to the 10 published tasks; five others remain hidden. The experiment selected Ornith because it showed looping on the test and was small enough to run on the available machine. The report says other local models either lacked the same looping behavior or could not be tested under comparable conditions.
“The edit repairs finishing, not legal knowledge.”
— Thorsten Meyer, ThorstenMeyerAI.com
Limits of the One-Run Comparison
The reported findings are preliminary and limited to one run per setting. The author identifies a small development set, a loop detector that changed during the experiment, and three graders from the same model family as limitations. Claude carried out the edit and Claude agents performed blind grading; the report says OpenAI’s GPT-6 Astra, running in Codex, independently reviewed the code, numbers and claims and found faults. That review is a check on the work, not a second experiment.
The comparison is also affected by different local setups and sampling settings. The article does not establish how often the edit would produce the same completion gain, whether the citation problems would persist across larger or hidden test sets, or whether other models would respond similarly. The score of zero is specific to this experiment’s published benchmark tasks and scoring method; the results do not establish a general score for legal AI systems.
The report makes the experiment’s runs and evaluation materials available through Lexassay, allowing readers to inspect the answers, grades and citation checks. Replication across repeated runs, using consistent hardware and sampling settings, would help separate the effect of the weight edit from variation in how the model was run. Testing the five hidden tasks could also help show whether changes generalize beyond the published examples, though the source does not report results on those tasks.
For now, the demonstrated outcome is narrower: the edited model finished more of the tested tasks in one same-setup comparison, but its legal score remained zero. No improvement in legal reliability has been established, and the article says the experiment is not legal advice.
Key Questions
What did the weight edit change?
The experiment measured an internal direction associated with looping and removed it at half strength from the output weights of 27 of Ornith 1.5 35B-A3B’s 40 layers. The report says there was no new training or legal data.Did the model become better at law?
The report found no improvement in its legal score: it remained at zero. The edited answers met 7 of 50 checked criteria, compared with 3 before editing, and included 55 wrong or invented citations.How many tasks did the edited model answer?
It answered all 10 published Lexassay tasks, compared with 4 on the same setup before editing. The report cautions that results varied across local setups and sampling settings.Was this the same method used by the MIT researchers?
No. The MIT report described an AI reasoning directly about another model’s weights. This experiment used directional ablation: it ran the model, measured internal signals, and removed a direction from selected weights.Can the result be treated as conclusive?
No. The report describes one run per setting, a small development set, a changing loop detector and grading limitations. The results need consistent, repeated testing before broader conclusions can be drawn.Source: ThorstenMeyerAI.com
