TY  - JOUR
DO  - 10.22454/PRiMER.2025.0129
VL  - 10
DA  - 2026/10/16
N2  - Introduction: Large language models (LLMs) can pass the US medical licensing exam and outperform residents on the family medicine in-training exam. The diagnostic performance of LLMs when applied to more complex medical cases, however, remains unknown.Methods: We evaluated the diagnostic accuracy of Claude (Anthropic), an LLM, on 20 case challenges published in The New England Journal of Medicine (NEJM). For each case, we presented Claude 3.5 Sonnet with multimodal input, including the text and all corresponding images from the cases. Claude was instructed to give an answer among multiple choices. We compared Claude’s diagnostic accuracy to human respondents on the NEJM website.Results: Claude 3.5 Sonnet diagnosed cases correctly about half the time, and at a significantly higher rate than NEJM readers (beta-binomial odds ratio 2.79; 95% CI [profile-likelihood] 1.09–7.01; P = 0.031). Claude was highly self-consistent in repeated exposure to questions, with perfect accuracy (100%) in nine cases and no accuracy (0%) in 10 cases, with one case at 90% accuracy. Human accuracy ranged from 9%–63% across cases. Claude’s performance was better on easier cases (ie, those with human accuracy >25%).Conclusions: Claude 3.5 Sonnet demonstrated significantly higher diagnostic accuracy compared to medical journal readers, though with notable variability in performance. Further research comparing consistency and reliability of artificially intelligent diagnostic capabilities to that of licensed physicians is needed.
PB  - Society of Teachers of Family Medicine
AU  - Litwin, Jack
AU  - Parente, Daniel J.
L2  - http://journals.stfm.org/primer/2026/parente-0129
L1  - http://journals.stfm.org/media/yo5npx1l/primer-2025-0129.pdf
TI  - Performance of Claude 3.5 Sonnet Compared to Human Readers of The New England Journal of Medicine Clinical Case Challenges
ER  -