Much of health tech’s work on diagnostic error happens at the point of care: decision support alerts, symptom checkers and, more recently, AI models that read notes and suggest what a clinician might have missed.
There’s also a case for looking earlier. Diagnostic reasoning is learned in training, and the habits a clinician builds there go with them into practice. It’s fair to ask whether software can help at that stage.
How big is the problem?
Bigger than it sounds. In its 2015 report Improving Diagnosis in Health Care, the National Academies of Sciences, Engineering, and Medicine concluded that most people will experience at least one diagnostic error in their lifetime.
The committee described improving diagnosis as a moral, professional and public health imperative. That’s the scale of the concern. It says nothing yet about whether training software helps, which is a separate question.
Where reasoning goes wrong
To see where training fits in, it helps to look at how diagnosis actually works.
In a 2006 review in the New England Journal of Medicine, Judith Bowen broke clinical reasoning into steps that can be taught. Gather the data. Summarize the patient in a short problem statement. Compare that summary with illness scripts, the mental models of how diseases typically present. Then settle on a diagnosis.
Each step is a place where reasoning can slip, and each is a place to give feedback. What data did the learner go after? How did they summarize the patient? Which illness scripts did they consider?
The scripts come from experience. Schmidt and Rikers, writing in Medical Education in 2007, described how clinical knowledge gets reorganized into illness scripts as students see more patients. Their advice to medical schools included bringing patient problems in early and building in time to reflect on cases with peers and experts.
The catch is exposure. A student on a clinical rotation sees the patients who happen to be there that week. Some conditions turn up often. Others a student may rarely or never see in person before graduating.
That’s the opening for software. Not to diagnose, but to add cases, earlier, with feedback on the reasoning steps. Whether a given tool actually does that is worth checking.
What training software can measure
The useful tools here are less about content and more about capturing reasoning. Case-based simulators can ask a learner to write a problem statement, rank a differential and justify the top choice before the case moves on. Platforms built around clinical reasoning cases for medical students can then let learners compare their problem representation and differential with an expert’s. Ask separately what faculty get to see.
That creates data a skeptic can actually use. A few measures worth asking for:
-
Accuracy on unseen cases, not on cases the learner has already practiced.
-
Calibration, meaning whether a learner’s confidence matches how often they’re right. Overconfident and wrong is the dangerous combination.
-
Differential breadth, or whether the right answer was on the list at all, even when it wasn’t first.
-
Change over time within the same cohort, not a one-off score.
None of these is exotic. A vendor that can’t report them should be able to explain what it measures instead, and why that’s a better guide to reasoning.
What to be skeptical of
Education technology has its own version of vanity metrics, and buyers should be wary of them.
Time spent in the app is not learning. Cases completed is not competence. Student satisfaction scores are worth having, but a tool that feels easy isn’t necessarily one that builds judgment. Useful practice often feels a little hard.
Be cautious, too, of tools that grade only the final diagnosis. A learner can reach the right answer by luck or pattern-matching on the case’s wording. If the tool records only the final answer, it can’t tell you much about whether their reasoning is improving.
The evidence so far
The evidence for virtual patients is encouraging but limited. A 2019 meta-analysis in JMIR by Kononowicz and colleagues pooled 51 trials and found virtual patients improved skills, including clinical reasoning, more than traditional teaching. The authors rated the certainty of the evidence as low to modest. It’s also worth noting what the review measured: learning outcomes in education settings, not what happened to patients later.
The harder evidence, a line from training cases to fewer diagnostic errors with real patients years later, is the thing to ask for. The studies above don’t show it. Ask vendors to keep demonstrated learning outcomes separate from claims about patient outcomes.
A reasonable position
Training software isn’t a fix for diagnostic error, and it shouldn’t be sold as one. What it can offer is a place where reasoning is made visible, practiced and measured before a real patient is involved.
If health tech is serious about diagnostic error, training deserves a share of its attention, held to the same test we’d apply to any clinical tool. Show the data, and show it on outcomes that matter.