🔍 Read the full analysis: 15 Questions, 42 AI Setups: A Test Of Legal Answers on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Lexassay, a test run by Thorsten Meyer AI, evaluated 42 AI setups on 15 legal questions covering German, US and EU law. Its results ranged from 8 out of 100 for a law-focused model to 79 for the top-scoring model, but the test is a limited spot check, and its operator discloses that Claude models graded answers. The findings point to citation errors and missed legal details as risks, not to a definitive ranking of legal AI tools.
Thorsten Meyer AI reported on November 4 that its Lexassay test scored 42 AI setups on 15 legal questions, finding results ranging from 8 out of 100 for a law-focused model to 79 for the highest-scoring model. The test also reported hundreds of citations that were invented or did not support the answers, a concern for readers who might rely on chatbots for legal deadlines or advice. Meyer disclosed that his organization runs Lexassay and that Claude agents graded the answers.
In the reported results, Claude Opus 5.5 scored 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. Meyer said the leading model fully met every criterion on only eight of the ten publicly available tasks. These are results from Lexassay’s scoring method, not a general measure of how well the models perform across legal work.
The model marketed by Z.ai for legal workflows, GLM-5.3 FlashX Legal, scored 8, with a score of zero on the hidden tasks. Lexassay’s citation review recorded 21 real citations, 25 it classified as misattributed and one it classified as invented. The test operator said the model gave no answer on two questions, including tasks about changes in German law and repair-cost claims. Its base model scored 3 under the same test.
Mistral Large 4 scored 10 in Lexassay, according to the report. Meyer said it answered about half of the substance correctly but lost many points over citations. He also noted that a separate Vals AI legal-agent benchmark, which supplies models with documents, ranked Mistral Large 4 eighth, ahead of Claude Opus 5.5 at 21st and GPT-6 Astra at 24th. The benchmarks test different conditions, so their rankings are not directly interchangeable.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Why Legal Citations Matter
The results matter because a confident-sounding legal answer can mislead if its citation is fabricated or misapplied, or if it misses a procedural detail. In the example used by Meyer, two German appeals faced different filing deadlines because Corpus Christi was a public holiday in Bavaria but not Berlin. The report says some models calculated one deadline incorrectly or placed a separate deadline a month too late.
Those are not merely wording errors: under German civil procedure, a late appeal can be rejected as inadmissible, according to the source article. The test therefore supports caution when using a chatbot for a filing date, a claim or correspondence with an authority. It does not establish that all legal AI is unreliable; it shows performance varied by model and task, and that answers require independent checking.
Lexassay reports 2,644 citations verified as real and 804 found to be invented or not saying what the model claimed across its published runs. Those are counts from the test, not a universal error rate: the source does not give a denominator suitable for calculating a proportion here. The figures also reflect Lexassay’s citation classifications and scoring rules.
Popular right now
TCL 65 Inch QM8L Series SQD-Mini LED QLED 4K Smart Google TV
Dyson Car+Boat Cordless Handheld Vacuum
Ninja Foodi 6-in-1 DualZone Air Fryer, 8 qt, Grey (DZ201)
As an affiliate, we earn on qualifying purchases.
How Lexassay Tested the Models
Lexassay asked each setup the same 15 questions: ten public and five hidden. The questions cover German, US and EU law, including deadline calculations, legal changes, court decisions and questions framed around false assumptions. The hidden-question scores are published without the answers, the report says, to make it harder for future models to reproduce them from online material.
Models answered without internet access or tools, testing what they could produce from the model alone rather than what a search-enabled legal assistant might find. Different reasoning-effort settings were run separately, meaning one model could appear more than once among the 42 setups. Each answer was compared with a gold answer using weighted criteria, and citations were checked against official databases.
The scoring rules deducted points for citations that existed but did not support the claim, as well as citations classified as invented. An answer without a legal-advice disclaimer or pointer to a lawyer also lost 5 percent. Meyer calls the exercise a spot check, not a strict benchmark: there was one answer per model and task, so small score differences may reflect noise rather than a stable gap in ability.
“It is a spot check, not a benchmark in the strict sense: one answer per model and task, so a few points of difference are noise.”
— Thorsten Meyer AI, the test operator
Limits of the Reported Scores
The results come from a test run by Thorsten Meyer AI, which also runs Lexassay. The operator says answers were graded blind by several Claude agents and discloses that a Claude model ranked first. That is a potential conflict of interest readers should weigh when interpreting the results. The source says public answers, gold answers and citation checks are published, allowing scrutiny of the grading.
The test does not show how the models would perform with web access, legal databases, supplied documents or help from a lawyer. It also provides only one answer per model and task, according to Meyer. The report does not establish how often the results would recur in repeated trials, nor whether performance would transfer to real client matters. Lexassay’s scores should not be treated as proof that one system is suitable for legal practice or that another is not.
The supplied material gives detailed results for some systems but is incomplete about the wider field: it ends during a discussion of GLM-5.3 FlashX Legal’s rank on another leaderboard. A full account of all 42 setups and their results cannot be confirmed from the material provided.
Checking Answers Before Acting
The source does not name a scheduled follow-up or a next testing date. Meyer says Lexassay publishes the test method and the public answers, gold answers and citation checks, which gives readers a way to review its scoring. The five hidden questions are treated differently: the report says only their scores are published.
For readers facing a real legal issue, the immediate next step is not to wait for another ranking but to verify any cited statute or ruling in an authoritative source and consult a lawyer about deadlines, contracts or official correspondence. The test’s findings are specific to its questions, settings and scoring approach; future results, including any repeat tests, would be needed to show whether model performance changes.
Key Questions
What did the 15-question test find?
Lexassay reported scores from 8 out of 100 for GLM-5.3 FlashX Legal to 79 for Claude Opus 5.5. Its citation checks also identified citations it classified as invented or as not supporting the answer. The results apply to the test’s questions and setup.
Does this prove which AI model is best for legal work?
No. The test operator describes Lexassay as a spot check, with one answer per model and task. It tested models without internet access or tools, while other benchmarks use different conditions and can produce different rankings.
Why does the test’s conflict-of-interest disclosure matter?
Thorsten Meyer AI runs Lexassay, and the answers were graded by Claude agents while a Claude model ranked first. That relationship should be considered when evaluating the results. The operator says it publishes the public answers, gold answers and citation checks so readers can examine the grading.
Can I use a chatbot to calculate a legal deadline?
The report advises against relying on a chatbot for a deadline. It describes a German example where a public holiday affected appeal dates differently in Munich and Berlin. For an actual deadline or legal matter, verify the applicable rules and ask a lawyer.
Source: ThorstenMeyerAI.com
