Microsoft’s new diagnostic AI outperformed generalist physicians on medical benchmarks. It asked the right questions, ordered various tests, reconsidered its assumptions and weighed the cost of reaching a diagnosis.

[For more news, click here]

For many years, people have been doing medical diagnosis by asking questions on Google and now AI. While these answers may not have been accurate in the past, AI is getting good at giving the right responses. When you give an AI model complete patient history, lab results, and a list of possible diagnoses, it performs impressively. However, the human analysis is largely missing, and that kind of diagnosis is only part of the process.

A doctor assesses a patient in real time while looking at their condition. When a patient comes in with an issue, the doctor has to decide what’s next, whether labs are necessary or a simple treatment is required.

Microsoft’s recent healthcare research has received interesting results. In a peer-reviewed study published in Science on October 22, researchers introduced a system designed to diagnose patients progressively rather than being handed all information at once. It is called the Microsoft AI Diagnostic Orchestrator or MAI-DXO; it coordinates multiple AI agents that behave like a panel of virtual clinicians. They propose possible diagnoses, challenge one another, decide which questions or tests can be useful, and revise their thinking as new evidence arrives. Read the research paper: Science and Microsoft AI

Moving Beyond the Medical Multiple Choice Test

One of the longstanding problems with AI in healthcare has been that it resembled less like a clinical practice and more like examining the results fed to it. Microsoft’s researchers wanted to create something closer to real clinical practice.

They created the Sequential Diagnosis Benchmark, or SDBench, in which the diagnostic agent receives only a short initial description. It must then decide whether to ask another question, order a test or make a diagnosis. A separate "gatekeeper" reveals information only when it is requested, while a judging system evaluates the eventual answer. The benchmark also keeps track of the estimated cost of the tests ordered. Microsoft’s SDBench benchmark and Microsoft GitHub

This is an important update, as simply knowing the right answer is not enough; it is deciding what information is worth obtaining next. If it orders fewer tests, it can miss or misdiagnose a serious condition. And if it orders too many tests, then the patient faces unnecessarily high costs, delayed diagnosis, and anxiety.

The researchers were trying to figure out a difficult question. Can AI navigate uncertainty? The study tested MAI-DxO against two different groups of cases.

The first consisted of 304 complex clinicopathological cases from the New England Journal of Medicine, the kind of difficult cases designed to challenge clinical reasoning. On these cases, Microsoft reported diagnostic accuracy of about 83% for MAI-DxO.

Generalist physicians scored approximately 22% without external resources and 36% when allowed to use references such as textbooks and the internet. Microsoft AI

Then the researchers moved beyond deliberately difficult teaching cases. They tested the system against emergency-department cases derived from MIMIC-IV-ED, a dataset based on de-identified patient records from Beth Israel Deaconess Medical Center. Microsoft reported around 80% accuracy for MAI-DxO compared with 47% for physicians on that evaluation. MIMIC clinical database

This result was far more important than the first. Hospitals typically see emergency-led, routine medical presentations rather than patients with rare conditions. So testing the system on emergency-department cases brings the research one step closer to real healthcare systems.

Accuracy vs Costs

It is still, however, far from a clinical trial, as accuracy was not the only measure. The system was also evaluated on how much its diagnostic work will cost. This was a big constraint. An AI, when given unlimited access to investigation, could theoretically improve its chances by simply requesting test after test. Get scans, more blood work, and more panels. This obviously does not make it a good doctor.

Microsoft therefore configured MAI-DxO to weigh the likely diagnostic value of additional information against its estimated cost. In one cost-optimized configuration on the NEJM cases, Microsoft says the system retained roughly 81% diagnostic accuracy while bringing the estimated work-up cost below half that of comparison systems. Microsoft AI

Healthcare systems do not simply need AI that can identify diseases; they need AI that understands when more information is valuable and when it is unnecessary. The difference between those things will eventually translate into fewer redundancies and faster decisions with lower costs.

Even though the attraction of this research is that AI beats doctors, it is not what happened inside the hospital. It did happen within certain benchmarks. The doctors in the comparison where individual generalists. The experiment did not establish that MAI-DXO is better than specialists, multidisciplinary clinical teams or physicians working with full acess to patients and all the information available in normal care.

The system was working in a controlled, text-based environment with historical cases. It was not examining patients, observing body language, dealing with incomplete information ot taking responsibility for treatment.

Microsoft acknowledges the distinction. Its own announcement says real clinical environments are the true test and that the technology still needs prospective evaluation. Reuters similarly reported that the next step will be to evaluate the system with patients at Beth Israel Deaconess Medical Center in Boston. Microsoft’s Research Announcement and Reuters

MAI-DXO still deserves some attention. It is not a giant medical model; it is an orchestration system. The architecture allows AI agents to perform different reasoning roles, debate possible explanations and collectively decide what information should come next. Microsoft says the approach improved performance across models from several major AI families rather than working only with one underlying model.

The breakthrough in this study is not that Microsoft has built a machine that can replace a doctor, as the evidence does not show that. It is that AI is beginning to tackle one of medicine’s harder intellectual problems: deciding what it needs to know before it can know what is wrong.

Related Articles

Exclusive: Why the GCC’s Next Healthtech Opportunity Is Bigger Than AI

Exclusive: An AI Just Passed a Video Turing Test. What Happens When You Cannot Tell Who is on the Call?

Abu Dhabi Signs AI Drug Discovery Agreement With Isomorphic Labs