🔍 Read the full analysis: The Benchmark Context Behind Z.ai’s Legal AI Near-Top Ranking on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai says its GLM-5.3-FlashX-Legal model achieved a 19.2% all-criteria pass rate on Harvey’s Legal Agent Benchmark. Vals AI’s archived leaderboard records 19.17%, seventh among 81 models; the score means the model met every rubric requirement on about one in five tasks, even though its individual-criteria pass rate was 92.5%.
Z.ai’s GLM-5.3-FlashX-Legal ranked seventh of 81 models on Vals AI’s archived Harvey Legal Agent Benchmark leaderboard, recording a 19.17% all-criteria pass rate. The result is a measure of tasks completed without a missed rubric requirement, not the share of legal work the model can perform or a guarantee that its answers are reliable.
Z.ai, the company formerly known as Zhipu, describes GLM-5.3-FlashX-Legal as a model optimized for professional legal workflows, including document understanding, legal reasoning, research, contract review and drafting. Its developer documentation says the model received legal-domain mid-training and post-training, with reinforcement learning across tens of thousands of task environments modeled on legal workflows. The documentation does not carry a publication date; Z.ai’s AutoClaw blog announcement is dated October 13, 2026, and Vals lists the release as October 12.
The benchmark result is reported as 19.2% in Z.ai’s materials and 19.17% in Vals AI’s archived leaderboard. Vals lists the model at $0.15 per test. The same leaderboard records a 6.67% all-pass score for the base GLM-5.3-Flash model. The difference is 12.5 percentage points, matching Z.ai’s stated improvement. Z.ai also calls the result state-of-the-art among open-source models, but the source material does not identify the comparison set, and it reports no published weights for the legal model.
The benchmark’s scoring rule explains why the headline percentage is low beside another reported metric. Harvey says a task passes only if every required criterion passes. Vals reports GLM Legal’s individual-criteria pass rate as 92.5%; that figure does not mean 92.5% of complete tasks succeeded. A task with one missed deadline, citation or other rubric item still fails the all-or-nothing task score.
Why the All-Pass Score Matters
The ranking offers a useful comparison, but the score answers a narrow question: how often did the model satisfy every listed requirement in a set of legal-agent tasks? It does not establish that the model is safe to use without review, that its work is legally correct in every jurisdiction, or that it outperforms every available model under other tests. Readers evaluating the claim should keep the task-level pass rate separate from the much higher criterion-level score.
That distinction matters for law firms and other organizations considering legal AI. A draft that misses one required issue may need substantial correction even if it gets most details right. The benchmark’s strict rule reflects that concern, while the high criterion rate may indicate that the model often satisfies many individual requirements. Neither measure alone establishes whether a human reviewer can efficiently identify and fix errors in real client work.
Cost is another point of comparison, not a measure of value or quality. Vals lists GLM Legal at $0.15 per test, lower than several higher-ranked entries shown in the archived table. That figure is a benchmark cost estimate, while Z.ai’s pricing page separately lists $0.37 per million input tokens and $1.25 per million output tokens. Actual deployment costs and review time depend on use and are not established by the leaderboard.
Popular right now
Vitamix 5200 Professional-Grade Blender, 64 oz. Container, Black
Breville Barista Express Espresso Machine, Black Sesame
LG 85-Inch QNED84B Mini LED 4K Smart TV (2026)
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Agents
Harvey released its Legal Agent Benchmark as an open-source benchmark on May 6, 2026. It contains 1,250 tasks across 24 practice areas, each asking an agent to produce a work product from an instruction and client-matter materials. Harvey says the tasks are graded against more than 75,000 expert-written, binary criteria covering items such as facts, conclusions, citations, recommendations, deadlines, amounts and formatting.
Harvey’s grading protocol calls a task complete only when all its criteria pass. Its example is a deal-team report that identifies eight of ten risks: under the benchmark’s rule, the two omissions prevent the task from counting as complete. Harvey has said that, in its initial baseline, the strongest all-pass rate among the frontier models it tested was 7.1%, and that those models completed less than 10% of tasks end-to-end in aggregate. That earlier result came from a different run and is not directly comparable with Vals’s leaderboard.
Vals AI’s leaderboard uses a private holdout of 120 tasks that it says mirrors the public benchmark mix, with internet access disabled. According to Vals, two AI judges grade submissions and the final score averages their task pass rates. Vals’s archived Version 1 table was updated October 18, 2026; a separate HLAB-Vals-1.1 page was updated October 30. The figures cited here are from the archived Version 1 table.
““On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%, an improvement of 12.5 percentage points over the base model, achieving state-of-the-art performance among open-source models in this evaluation.””
— Z.ai
What the Ranking Cannot Establish
The leaderboard establishes a result under one benchmark setup, not how the model will perform across law firms, jurisdictions, practice areas or live client matters. The source material does not provide the underlying task-by-task results, a breakdown by practice area, or enough detail to assess how often particular error types occurred. It also does not explain which models Z.ai included in its claim of state-of-the-art performance among open-source models.
A small, non-benchmark spot check described by ThorstenMeyerAI.com reported that the legal model cited rulings the reviewer could not find. In the same four-task check, OpenAI and Claude models did not invent citations, while two Claude models cited a real ruling for the opposite of its holding. This limited exercise is not evidence of comparative error rates and should not be treated as a benchmark result. The source also says the Z.ai pages it reviewed presented outputs as drafts for legal professionals and carried no “not legal advice” notice.
It remains unclear what safeguards Z.ai offers for client data, what rules apply to data processed through its services, and how the company recommends firms verify citations and work product before use. The benchmark does not answer those procurement and legal-review questions.
What Buyers Should Verify
Z.ai recommends running the model as an agent connected to MCP services for laws, cases and company data. That setup makes deployment details relevant: firms will need to check what information is sent to the model and connected services, how it is handled, and what contractual or technical protections apply. The source material does not specify those terms or announce a next benchmark evaluation.
For now, the concrete next step for readers is to distinguish the dated Vals leaderboard result from Z.ai’s broader product claims. Firms assessing the model can examine its performance on tasks resembling their own work, test how it handles citations and omissions, and set review procedures before any draft leaves the organization. The available information does not establish a release timetable beyond the announced model and application-based AutoClaw beta, nor whether the leaderboard will be updated with further results.
Key Questions
What does GLM-5.3-FlashX-Legal’s 19.17% score mean?
It means the model met every required rubric criterion on 19.17% of the benchmark tasks in Vals AI’s archived leaderboard. A task that misses even one required criterion does not count as a pass.
Why is the 92.5% criteria pass rate higher?
The 92.5% figure measures individual criteria, while 19.17% measures whole tasks. A model can pass most criteria on a task but still fail that task if one or more requirements are missed.
Was GLM Legal seventh among all legal AI systems?
It was listed seventh of 81 models on Vals AI’s archived Version 1 leaderboard for Harvey’s benchmark. That ranking applies to that leaderboard and scoring setup, not every legal AI system or evaluation.
Does the result prove the model is ready for legal work?
No. The benchmark reports performance on a defined set of tasks; it does not establish reliability in live client matters or remove the need for professional review. The source material also leaves data-handling and deployment terms unclear.
Source: ThorstenMeyerAI.com
