Predictive Value of In-Service Exams

In the September 2022 issue of Family Medicine, I read Torres et al’s “The Predictive Value of the Residency AOBFP In-Service Exam, Produced and Administered by ACOFP,” ¹ and I enjoyed it greatly. I congratulate the authors for having made transparent many of the procedures and results of the American Osteopathic Association family medicine examinations. There is tremendous value in this type of conversation continuing. I was glad to see that their specificity and negative predictive value were much lower than their sensitivity and the positive predictive value. The passing predictions from the In-Service Examination (ISE) were strongly associated with actual passes on the certification exam. Also, failing predictions were much weaker, which implies that residents and program directors are using ISE or ITE feedback in a timely way to prevent residents from failing their first attempt on the certification exam. This is good news for family medicine!

I would also like to point out two minor errors in the article. First, the American Board of Family Medicine (ABFM), not the American Academy of Family Physicians, produces and owns the ABFM’s In-Training Examination (ITE). The second is that, in the results section, the correlations between ISE scores and the AOBFP-CE (American Osteopathic Board of Family Physicians’ certifying exam) reported in the text do not match the correlations in Table 1.

Whichever set of correlations is correct, these correlations were just a little lower than what I and my coauthors found in our 2016 ABFM paper.² Because we used very similar methods, we can attempt to explain differences in the outcomes based upon identifiable differences in conditions. I mention this because you pointed out that the American College of Osteopathic Family Physicians and AOBFP are responsible for the ISE and the AOBFP-CE, respectively. You may be able to improve the correlation between the two exams by ensuring that, across both examinations, the same blueprint is being used, the same item writing procedures are in place, and both examinations are equated onto the same scale, preferably using a Rasch model like the AOBFM-CE did. This is much easier to do if a single organization owns both examinations. It avoids duplication in the item writing and psychometric efforts, and it eliminates the need for legal agreements between the two organizations. This is easy to ask for, but difficult to make happen.

The work you are doing is important because ITEs and ISEs should identify residents who need a little extra help in a timely manner. I cannot wait to see what your next steps will be. Well done.