🔍 Read the full analysis: Making Sense Of Microsoft’s AI Beats Doctors Claim on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A peer-reviewed Microsoft study reports that its MAI-DxO system diagnosed selected complex medical cases more accurately than generalist physicians in a text-based simulation. The results do not show how the system performs with real patients, specialists or ordinary cases, and the authors have not yet had the results independently reproduced.
Microsoft’s MAI-DxO system scored higher than generalist physicians on selected medical diagnostic cases in a study published in Science on October 22, 2026. The peer-reviewed paper reports 83.2% accuracy on 304 complex case records, compared with 36.1% for physicians using textbooks and the internet and 22.3% without those resources. The test used historical, text-only cases in a simulator—not live patients—so the result does not establish that the system is safer or more effective than doctors in clinical care.
The paper, titled Sequential diagnosis with language models, also reports results on 325 emergency department cases: MAI-DxO reached 80.3% accuracy, compared with 46.8% for the physician comparison group. These are results reported by Microsoft’s research team. All 15 authors are listed with Microsoft affiliations, including Microsoft AI chief executive Mustafa Suleyman. Publication in a peer-reviewed journal means the study underwent scientific review; it does not mean researchers outside Microsoft have reproduced its findings.
The benchmark attempts to model diagnosis as a sequence of decisions rather than a one-question exam. Each case begins with a short summary, and the system or physician must request additional information, such as a test result, from a simulated case file. The study scores both the final diagnosis and the cost of obtaining information. MAI-DxO uses a language model as a panel of virtual physicians that proposes diagnoses, selects follow-up questions or tests and challenges its own reasoning across multiple rounds.
The comparison has important limits. The physician group consisted of primary care doctors and hospital generalists working individually; the study did not include specialist teams. The cases were selected as difficult teaching examples, and the simulation supplied information through text rather than physical examinations or real-world imaging. Microsoft’s published figures also show GPT-5 alone scored 78.9% on the complex cases, close to MAI-DxO’s 83.2%. That suggests much of the measured performance came from the underlying model, while the orchestration system added a smaller increase and lowered estimated testing costs.
Infographic 01 · The scoreboard
83% vs. 36%: the number, and who it was measured against.
Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.
Hard cases (NEJM)
Diagnostic accuracy, 304 case records
Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.
Who was compared
The fine print behind “beats doctors”
- Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
- Working alone: no colleagues; in the preprint also no search engines or language models.
- Text-only, historical cases: no real patients, no physical exam, no imaging.
- Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
- Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.
Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI
What the Scores Mean for Care
The results matter because they put a concrete benchmark behind a high-profile claim about AI-assisted diagnosis. If similar performance holds in clinical settings, a system that helps clinicians consider possible diagnoses and choose tests could affect how quickly patients receive answers and how medical resources are used. But this study does not establish that outcome: it evaluates performance on a constructed benchmark, not changes in patient health, clinician workload or hospital spending.
The study’s cost estimates also should not be read as observed savings. According to the source account, test prices were estimated using billing codes and a 2023 U.S. hospital price list, with a fixed physician-visit charge. The researchers describe these as approximations. A simulated price comparison cannot show what a hospital would actually spend after accounting for staffing, follow-up care, errors or the cost of integrating a system into clinical practice.
For patients and health systems, the immediate takeaway is that diagnostic accuracy on selected cases is not the same as proven clinical benefit. A tool might identify a correct diagnosis in a benchmark yet still perform differently when information is incomplete, symptoms are ambiguous or a patient has multiple conditions. Evidence from prospective testing will be needed before the reported scores can support decisions about routine use.
AI diagnostic tools for healthcare
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
How Microsoft Tested the System
Microsoft’s Sequential Diagnosis Benchmark is designed to test the process of gathering information as well as the final answer. A “gatekeeper” model holds the full case record and supplies details when requested. In the earlier preprint, missing test results could be generated synthetically by the gatekeeper, rather than treated as unavailable. That design is intended to avoid revealing whether a diagnostic path is wrong simply because a result is absent, but it means some benchmark information is simulated.
The source account says the Science paper followed a June 2025 preprint that reported a larger gap using an earlier system based on OpenAI’s o3. The peer-reviewed paper’s numbers differ, and the final published study adds a physician comparison using textbooks and the internet. According to the account, the preprint’s physician testing used 56 held-out cases, while the AI was assessed across all 304 complex cases. The exact methods in the final paper may differ from the preprint; the full Science text is behind a paywall, limiting independent review of every detail in the source account.
Microsoft’s blog described the findings as showing the system was 1.7 times as accurate as physicians on emergency cases, more than twice as accurate as physicians with reference materials on complex cases, and nearly four times as accurate as physicians without them. Those ratios reflect Microsoft’s reported benchmark results, not a demonstrated advantage in patient care. The researchers also used a model to grade diagnoses against published answers; physician re-grading of a sample showed good, but not perfect, agreement, according to the source account.
“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”
— Microsoft AI chief executive Mustafa Suleyman, speaking to TechCrunch
As an affiliate, we earn on qualifying purchases.
Limits the Trial Has Not Settled
It remains unknown whether MAI-DxO can match its benchmark performance with consenting patients receiving ordinary care. The reported tests did not include physical examinations, real imaging workflows or the full range of cases seen in routine practice. The comparison also leaves open how the system would perform alongside specialists or multidisciplinary teams, rather than individual generalists.
The case mix creates another limitation. The source account says the NEJM cases contained no healthy patients and no harmless explanations, so the study could not measure false alarms. A system tested on a set of unusually difficult cases may not make the same trade-offs when common, mild conditions are included. It is also not yet clear how often the system might recommend unnecessary tests or miss a diagnosis in a broader patient population.
Independent replication has not been reported in the supplied material. Other unresolved questions include how the final Science methods compare in detail with the earlier preprint, whether its estimated cost advantage would persist in hospitals, and how clinicians would use or challenge the system. Peer review evaluates a paper; it does not settle those questions.
As an affiliate, we earn on qualifying purchases.
A Prospective Patient Study Ahead
The next stated step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source account, consenting patients will interact with the system alongside their normal care. That setting should provide evidence closer to clinical use than a retrospective text benchmark, although the supplied information does not specify the study’s start date, size, design or outcome measures.
Readers should watch for whether the study reports not only diagnostic accuracy but also false alarms, missed conditions, test use, patient outcomes and how clinicians respond to the AI’s suggestions. Until those findings are available, Microsoft’s Science paper supports a narrower conclusion: MAI-DxO performed well on the particular simulated cases tested, while its value and risks in patient care remain to be established.
As an affiliate, we earn on qualifying purchases.
Key Questions
Did Microsoft prove its AI is better than doctors?
No. The paper reports that MAI-DxO scored higher than a group of generalist physicians on selected historical cases in a simulation. It did not test whether the system produces better outcomes for real patients.
What accuracy did MAI-DxO report on complex cases?
Microsoft’s study reports 83.2% accuracy on 304 complex case records. The physician comparison scored 36.1% with textbooks and internet access and 22.3% without those resources.
Were specialists included in the comparison?
No specialist teams were included, according to the source account. The comparison was with primary care doctors and hospital generalists working individually.
Has the system been tested with patients?
The reported benchmark used historical, text-only cases rather than live patients. A prospective study at Beth Israel Deaconess Medical Center is planned, with consenting patients interacting with the system alongside their normal care.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
