Performance of Claude 3.5 Sonnet Compared to Human Readers of The New England Journal of Medicine Clinical Case Challenges

Large language models (LLMs) are artificially intelligent (AI) systems that use artificial neural network machine learning on large sets of data to statistically describe language. Applications of AI to family medicine and medical education are being widely discussed.^1-41 LLMs are largely black boxes: Inspection of the inner workings has only limited ability to explain observed outputs. Thus, LLMs must be evaluated based on the quality of their outputs alone. Fortunately, medical educators have long experience in evaluating black-box neural networks: using exams to evaluate human brains.

Indeed, LLMs have done well on medical exams. GPT-4 (OpenAI) passed the US medical licensing exam (USMLE)⁴² and scored higher on the family medicine in-training exam (ITE) than third-year residents.²⁰ While this is exciting, the USMLE and ITE are artificial scenarios that are, by design, contrived to eliminate the uncertainty of real-world practice. Less research exists on how well these models can perform in clinical settings with higher complexity.

The public conversation around LLMs was spurred by the release of ChatGPT (OpenAI) in 2022.^43,44 However, many other models exist. A recent comparison between Google Bard, Perplexity AI, ChatGPT, and Claude (Anthropic) found that Claude scored higher in relevance and completeness when analyzing medical cases.⁴⁵ In light of these new studies, we therefore focused on the performance of Claude 3.5 Sonnet.

Here, we evaluated the diagnostic capabilities of Claude on complex cases drawn from the published literature,^46-65 using an approach previously applied to ChatGPT.⁶⁶ We hypothesized that Claude would significantly outperform human readers in answering multiple-choice questions based on published, real-world, complex cases.

We retrieved all 20 freely available (at time of curation) The New England Journal of Medicine (NEJM) case challenges released between 2020 and 2024.^46-65 Each case, based on real patient encounters, included a history and physical with pertinent imaging and/or laboratory results. Readers selected the most likely diagnosis from a list of six answer choices.

We presented Claude 3.5 Sonnet with a standardized prompt (Appendix), followed by the text of the case and any related images in JPEG format, excluding figure/table captions but including table footnotes. The prompt instructed the AI model on how to approach the task, emphasizing careful reasoning and selection of a single most likely diagnosis from the options provided.

Claude’s output is partially nondeterministic: The same input might result in different output. For any given question, Claude might always select a correct answer, might always select an incorrect answer, or might answer correctly only some of the time. We therefore presented each case to Claude 10 times and calculated the accuracy rate across these trials (20 cases × 10 repetitions). We performed these analyses between September 24, 2024, and October 20, 2024. We calculated Claude’s per-case and overall accuracy.

We compared Claude’s accuracy with the accuracy of NEJM readers, as reported on the NEJM website, using beta-binomial regression with the logit of human accuracy included as a per-case offset.

The University of Kansas Medical Center Institutional Review Board determined this study was not human subjects research.

Sample output is shown in the Appendix. Claude 3.5 Sonnet made the correct diagnosis in 99 of 200 (49.5%) trials. Within cases, Claude was almost entirely self-consistent: 10 cases having 0% accuracy, one case having 90% accuracy, and nine cases having 100% accuracy (Figure 1).

A median of 7,576 (IQI: 5,751–9,193) human readers responded to each of these cases on the NEJM website. Human respondents had an overall accuracy of 27.4%. Accuracy on individual cases was—as with Claude—highly variable, ranging from 9%–63% across cases. Claude’s accuracy was significantly higher than human readers (beta-binomial odds ratio 2.79; 95% profile-likelihood CI 1.09–7.01; P = 0.031).

Among cases humans answered with greater than 25% accuracy (“easier”), Claude achieved 74.1% (89/120) accuracy, as compared to only 12.5% (10/80) accuracy on “harder” cases.

We hypothesized that Claude 3.5 Sonnet would have superior diagnostic capabilities, as compared with human NEJM readers, when presented with complex clinical case challenges; this hypothesis was borne out. First, we concluded that this hypothesis suggests that LLMs may help physicians approach real-world situations that include diagnostic uncertainty. However, Claude’s performance was dependent upon case difficulty, struggling with questions humans likewise find difficult.

Second, AI is not a diagnostic panacea: Claude generated incorrect solutions in half of the cases. Our findings are concordant with similar studies evaluating ChatGPT-4.⁶⁶ AI systems should be viewed with open-minded skepticism unless and until their reasoning is reliably correct.

Third, despite Claude’s nondeterministic output, its answers were almost entirely consistent across trials (199/200 trials). Claude’s responses were, thus, repeatable, and asking Claude once for an opinion seems sufficient, rather than prompting multiple times.

These conclusions are nevertheless preliminary due to limitations inherent in this study. Most importantly, we cannot assess the medical background of the NEJM readers and thus cannot extrapolate the data in comparison with licensed physicians, who would be the gold-standard reference population. Second, this study evaluated diagnostic accuracy on multiple-choice questions; given that these cases presented six multiple-choice answers, the model and the readers had a 16.7% chance of guessing correctly. Real-world applications of LLMs would not have multiple choice.

While diagnostic accuracy is an important part of a physician’s job, physicians require other skills to practice medicine. Neither AI nor journal readers were asked to take a thorough history or perform a detailed physical examination. Furthermore, all relevant labs and tests had seemingly already been ordered, and explanations were provided for many of the radiological findings. Real medical practice requires deciding not only how to react to already-collected information but also deciding what information (if any) to collect next.

Future studies comparing Claude to licensed physicians on complex cases would be worthwhile. Likewise, to ensure the best models are used, future studies should assess not only a specific LLM’s diagnostic capabilities in comparison to physicians but also in comparison to other LLMs. Furthermore, LLMs are only one component of artificial intelligence, and exploring how the integration of various tools can be used in clinical practice will be important.

Litwin J. Claude 3.5 Sonnet Medical Case Review: Analyzing the Diagnostic Capabilities of Claude 3.5 Sonnet on Complex Clinical Cases. Oral Presentation at KUMC Student Research Forum, April 8–9, 2025, Kansas City, KS.