Impact of the Introduction of the Clinical Science Mastery Series on Family Medicine NBME Subject Examination Performance

Many medical schools assess student clinical knowledge using the National Board of Medical Examiners (NBME) subject examinations. While the NBME Clinical Science Mastery Series (CSMS) existed for all other clinical subjects, Family Medicine CSMS self-assessments were first introduced in September 2017. The CSMS self-assessments consist of examination questions that were previously part of the NBME examinations. As such, they provide students with realistic, representative practice questions as well as immediate performance feedback. To further assess the utility of various study tools available to our students, this study investigated the impact of the CSMS self-assessment for family medicine on the NBME family medicine subject examination performance.

Multiple variables impact NBME subject examination performance. Various studies have investigated scoring trends and student preparation for NBME examinations. Several studies demonstrated that mean scores on NBME examinations improve as the clinical year progresses1-3 and that clerkship order impacts scores on the NBME surgery4 and family medicine5 subject examinations. Students prepared for the NBME psychiatry subject examination most frequently by using preparation books and next with practice questions (84.6% of responding students).6 fmCASES National Examination is a helpful formative assessment tool for students beginning their family medicine clerkship.7 However, use of MedU (now Aquifer) virtual cases did not impact NBME scores.8 Performance on CSMS self-assessments and clinical science subject examinations was related.9 Students used CSMS self-assessments both as learning tools and as instruments to gauge their clinical science knowledge.9 As such, we hypothesized that student performance on the NBME family medicine subject examination would show improvement after the introduction of Family Medicine CSMS self-assessments.

As part of the continuous process of reviewing and evaluating the family medicine clerkship at our institution, we noted that students reported regular use of the CSMS to prepare for NBME subject examinations, across all clerkships. However, prior to September 2017, there was no Family Medicine CSMS for students to use to prepare for the examination. In an effort to identify whether the CSMS impacts clinical subject examination performance, we compared student NBME examination scores taken after the CSMS release date to examinations taken prior to the release, using the academic year percentile ranks tables provided by NBME to match for the time of year. As part of their third-year clinical competency requirement in family medicine, students at our institution take the NBME Family Medicine Clinical Subject examination on the last day of the 8-week family medicine clerkship. There were no major changes made to the curriculum in the family medicine clerkship that might have affected scores.

All 185 participants’ scores for this study were from third-year medical students in their family medicine clerkship at the Herbert Wertheim College of Medicine. All students are required to complete and pass the end-of-rotation NBME Family Medicine Clinical Subject examination, worth 25% of their overall clerkship grade.

The student groups are comparable with the pre-MSCS group with an average MCAT score of 30, GPA of 3.67, and biology, chemistry, physics and math (BCPM) GPA of 3.60 and students in the MSCS group had an average MCAT score of 30, GPA of 3.63, and BCPM of 3.54. This study was reviewed and approved by the Florida International University Institutional Review Board.

The NBME Family Medicine CSMS self-assessment is a web-based, self-paced, 50-item, multiple-choice examination that became available in September 2017 and is self-paced. NBME has multiple forms available and they charge $20 for each self-assessment. Test takers are provided with a feedback report that shows their performance by USMLE content area; incorrect questions are identified, and a score interpretation guide estimates a how well the test taker is expected to perform in the actual subject exam.

NBME scores all family medicine exams using an “Equated Percent Correct Score,” defined as the percentage of items in the total content domain that would be answered correctly based on an examinee’s proficiency level. The subject examination scores are equated across test administrations and are statistically adjusted for variations in test form difficulty.10 The Equated Percent Correct Scores were analyzed in this study.

Since subject examination performance improves throughout the academic year, NBME uses quarterly percentile tables to account for score improvements. In order to have matched groups based on the NBME tables, we attempted to collect the same number of examinees per quartile for the pre-CSMS (n=95) and the CSMS (n=90) conditions. For example, there were 21 CSMS quarter 1 scores and 22 pre-CSMS quarter 1 scores. Pre-CSMS examinations were delivered between November 2016 and August 2017, and CSMS examinations were delivered between September 2017 and June 2018.

Statistical analysis was conducted using SAS statistical software (SAS version 9.4, Cary, NC). Independent t-test analysis was used to compare the percent correct scores on the end-of-rotation NBME family medicine clinical subject examination between the students who took the NBME family medicine subject examination before (n=95) the availability of the CSMS self-assessment examination and those who took the clinical subject examination (n=90) after the release of the CSMS Family Medicine self-assessment. The effect size between the two groups was analyzed using a Cohen d analysis to determine the magnitude of the performance differences between the pre-CSMS and the CSMS groups.

Table 1 shows the pre-CSMS and CSMS distributions which were sufficiently normal for the purposes of conducting a t-test (ie, skew <2.0 and kurtosis <9.0).11 Table 1 also outlines the results, showing significant differences in clinical subject exam scores prior to the availability of the CSMS and after the availability of the CSMS. The independent samples t-test was associated with a statistically significant effect (t[183]=3.28; P=.001). Hence, CSMS was associated with improved clinical subject examination scores. The associated effect size (Cohen d) for this analysis was estimated at 0.48, near Cohen’s12 convention for a medium effect (d=0.50). The means and the 95% confidence intervals are shown in Table 1.

The NBME subject examination is an objective measure of student knowledge, thus increases in this score indicate increased student knowledge. Also, subject examination scores are a significant percentage of overall clerkship grades, including pass, near honors, and honors designations, which impacts rankings and residency applications.

These outcomes suggest that the introduction of NBME CSMS family medicine self-assessments improved student performance on the end-of-rotation clinical subject examination. Our results indicate that the CSMS serves as a useful study tool for family medicine NBME subject examination preparation. Though we did not specifically identify individual student use of CSMS self-assessments, our results affirm students’ anecdotal feedback that the CSMS self-assessments are valuable study resources, both supplementing their study and orienting them to NBME examination content and format.

Several factors may be involved in the successful use of CSMS self-assessments as study guides. First, the questions are derived from former NBME questions and therefore likely accurately represent the types of questions that students will see on the actual examination. Second, student self-assessment using the CSMS serves as a readiness checkpoint and may focus student studying. The CSMS self-assessment provides a coaching report that guides the student to identify their strengths and weaknesses, a scoring conversion table, performance by content area, and the ability to review incorrectly answered questions. The impact of the self-assessments may vary depending on when in the clerkship they are taken; students may opt for early in the rotation to improve content awareness, or later in the clerkship to assess knowledge. This may enable students to better gauge their family medicine knowledge level, leading to more directed and intensive studying.

Another consideration is cost. CSMS self-assessments cost $20 each; over the course of the multiple NBME examinations taken during medical school, this expense may become a significant barrier for some students. During the last 2 weeks of the first rotation of the clinical year, our school pays for students to take a formative CSMS, serving as an orientation and readiness checkpoint, during their first clinical rotation. The examination is administered using timed, formal testing conditions. In an effort to improve preparation for all students and guide students to study relevant subject matter for their NBME subject examinations, medical schools may consider funding these self-assessments.

This study has limitations. First, the family medicine CSMS self-assessment has only been available since 2017, thus limiting the sample size for comparing student performance after the self-assessment was introduced. Another limitation is that we did not have an accurate figure of which or how many students actually used the Family Medicine CSMS self-assessment, and thus could not follow their testing trends specifically.

Our students take Step 1 at the end of the third year; while one study notes that Step 1 performance can identify students at risk for poor performance on the NBME subject examinations, this is not relevant at our institution.13 Future studies may compare student NBME performance from the same clerkship block who did and did not use the CSMS self-assessments. Further research may also assess how students use the CSMS self-assessments as a study guide and formative feedback tool to better prepare for the subject examination.