AI medical diagnosis: regulators are chasing an evidence base that keeps moving

Medical AI regulation is being drafted against a moving target. The same class of models is being sold as a diagnostic assistant, tested as a benchmark champion, and audited as a privacy risk, often within months of each other.

Consider the evidence regulators would have to weigh. Microsoft announced its AI Diagnostic Orchestrator (MAI-DxO) on 30 June 2025, according to GeekWire. The company reported 85.5% correct diagnoses on cases drawn from the New England Journal of Medicine, compared to 20% for 21 experienced physicians from the US and the UK. WIRED, reporting on the same experiment on 1 July 2025, described an accuracy of 80% for the tool against the doctors' 20%. It also noted a 20% reduction in costs from choosing cheaper tests. Both figures rest on 304 recent NEJM cases assembled into what Microsoft calls the Sequential Diagnosis Benchmark.

The gap between 85.5% and 80% is not a rounding quibble. It is the kind of discrepancy a regulator must resolve before writing a threshold into law.

Microsoft's own framing was careful. GeekWire noted that the research paper behind the tool had not yet been accepted by a scientific journal. The company said the tool would need testing against more commonplace ailments, clinical validation for safety and performance, and regulatory approval before deployment. Bay Gross, Microsoft AI's vice president of health, called it "a proof-of-concept" in a LinkedIn post. Mustafa Suleyman, CEO of Microsoft AI, was less measured: "We're taking a big step towards medical superintelligence," he wrote on LinkedIn.

The doctors were not allowed to use tools

Independent researchers quoted by WIRED raised a methodological problem that any regulator would hit immediately. David Sontag, a scientist at MIT and cofounder of Layer Health, said the findings should be treated with caution. The doctors in the study were asked not to use additional tools, which may not reflect real practice. Eric Topol of the Scripps Research Institute called the report impressive for tackling complex cases. He said showing AI could in theory cut costs was novel. Both agreed the next validation step would be a clinical trial comparing the system with real doctors treating real patients.

That trial has not happened. Microsoft has not decided whether to commercialise the technology, WIRED reported. However, an executive said the company could integrate it into Bing to help users diagnose ailments.

The other half of the picture matters more for anyone writing patient-facing rules. Research published in JAMA Network Open in April 2026, led by Harvard medical student Arya Rao, tested 21 leading off-the-shelf AI models on 29 standardised clinical vignettes. The models did well on final diagnosis, correct 91% of the time when handed a full portfolio of medical information. Early differential diagnosis, the stage where clinicians rule conditions in and out under uncertainty, failed in more than 8 out of 10 cases.

"Every model we tested failed on the vast majority of cases," Rao told The Register. "That's the stage where uncertainty matters most, and it's where these systems are weakest."

Rao also told The Register that failure in the paper did not mean the AI completely missed. It only meant the system did not produce a fully correct answer. Measured as raw proportion correct, the models scored between 63% and 78%. That is a far better number than the headline failure rate. It is the kind of nuance that tends to disappear by the time a statistic reaches a consultation on regulation.

Dr Marc Succi, a radiologist at Massachusetts General Hospital and a coauthor, told The Register that higher success rates on final diagnosis should not be reassuring. "Real clinical reasoning starts earlier, when ambiguity is highest, and that is exactly where they remain weakest," he said. "Even if you get to the final answer eventually, the wrong differential can result in delays in care, unnecessary procedures with complications, high costs, and much more."

Privacy audits measure the wrong thing

A third strand that regulators have been slower to fold in is what the training data itself gives away. German researchers reported in a Nature paper published on Wednesday 24 June 2026 that discriminative medical AI models are susceptible to membership inference attacks. These are queries designed to establish whether a particular patient's record was in the training set. The Register reported that the team analysed seven medical AI datasets covering images, ECG records, and general electronic health records. They found individual patients could be identified with what the researchers called "near-perfect attack success."

The finding cuts against current audit practice. "The fact that MIAs can achieve near-perfect success rates for individual patients is not adequately captured by the standard evaluation protocol, which measures attack success in aggregate across records," the researchers wrote. Their conclusion is that reporting standards for AI privacy audits need to change. Differential privacy frameworks, designed to give mathematical guarantees that training data stays anonymous, should be used where necessary.

Underrepresented patients are easier to expose than others. Race, insurance status, sex, the imaging protocol used, and certain disease statuses can all function as outliers that make an individual identifiable, according to the paper. "Generally speaking, privacy risks from MIAs become more severe as a model's training cohort becomes more specific," Moritz Knolle, who leads the AI in Healthcare and Medicine chair at the Technical University of Munich and is lead author, told The Register. He raised the prospect of membership in a training dataset revealing that someone has a dormant genetic condition such as Huntington's disease, or that they attended a specialised treatment clinic.

Knolle was clear about the limits of the attack. An attacker needs access to a target data point, though the paper shows partial access is enough, contrary to what was previously believed. The datasets used in the study were anonymised. "The attacker would simply need access to someone's blood test results, or part of these results" to infer inclusion, he said.

What this means for the rules

Three practical problems fall out of this for anyone drafting medical AI regulation.

• Benchmarks are not clinical evidence. Microsoft's 85.5% and 80% figures come from retrospective NEJM cases, with physicians barred from consulting colleagues or resources. Both Sontag and Topol told WIRED that a clinical trial is the next required step.
• Headline accuracy hides the failure mode. A model can reach 91% on final diagnosis and still fail early differential diagnosis in more than 80% of cases, as Rao's team found. Rules written around a single accuracy number would miss exactly the stage where the study says harm originates.
• Privacy auditing measures the wrong unit. Aggregate attack success rates, the current standard, do not capture near-perfect success against individuals, per the Nature paper.

There is also the question of who is doing the testing. The Microsoft benchmark was constructed by Microsoft, from Microsoft's selection of cases, and the paper had not cleared peer review at the time of the announcement. The JAMA Network Open study tested models off the shelf, without vendor involvement, and produced a much worse result. The Nature privacy paper tested seven datasets and found an audit gap.

None of this means medical AI cannot be regulated. It means the numbers regulators will be handed come with provenance attached, and the provenance matters as much as the number.

Knolle's stated aim was modest. "I hope that the medical AI community will start to take privacy risks seriously and that risk mitigation techniques are used in situations where they are necessary," he told The Register. He also noted that many successful membership inference attacks represent a small or negligible privacy violation. This applies in cases where models are trained on large, general populations that include both healthy and diseased individuals. The risk scales with how narrow the cohort is, which in medicine often means the sickest and the rarest patients.

Suleyman has said the company will spend the next couple of years proving these systems out in the real world. That work, not the benchmark, is what any serious rulemaking will eventually have to assess.