Knowledge Self-Assessment Engagement and Family Medicine Board Examination Outcomes

Some physicians question the importance and role of high-stakes summative assessments (assessments of learning with comparisons to peers and minimal performance standards required to achieve or maintain a credential) after graduate training. ^1-3 They argue that continuing medical education (CME) or formative assessment (feedback without minimal performance standards designed to guide self-directed learning) is sufficient. ⁴ While some CME formats are associated with changes in clinical practice, known limitations exist to relying on CME alone for maintenance of knowledge. These include CME selection based on interest rather than data-defined gaps ⁵ and decay of knowledge and quality of care over time. ^{6, 7} American Board of Family Medicine (ABFM) certification signifies that family physicians possess knowledge across the breath of the specialty, even though some family physicians may narrow their scope of practice over time. Thus, ABFM, as well as all American Board of Medical Specialties (ABMS) boards continue to incorporate summative knowledge assessments as part of the continuing certification process. ⁸

Studies on the relationship between formative and summative assessments in trainees have had mixed results. Some studies found no correlation, ^{9, 10} while other studies found a positive association between serial formative assessments and summative assessment performance. ^11-13 However, evidence on the assessments’ relationship in practicing health care professionals is sparse. ¹⁴

Research has shown that assessment consequences (stakes) impact performance and learning. ^15-17 Previous work has shown that physicians perform better, albeit with lower confidence, on higher-stakes examinations compared with lower-stakes self-assessment. This evidence suggests that higher-stakes examinations may be a better reflection of physicians’ fund of knowledge. ¹⁴

Physicians may concentrate less or put less effort into low-stakes formative self-assessments, given the lack of consequences for low performance. The degree of engagement in formative assessment may influence the association between formative and summative assessment performance. ¹⁸ Engagement can be categorized into behavioral, cognitive, and affective dimensions. ^{19, 20} Behavioral engagement is observable and includes attendance, persistence, paying attention, asking questions, and avoiding disruptions. Cognitive engagement is deeper, including efforts to process and understand concepts and retrieve previous knowledge on which to build. Affective engagement includes emotional reactions, general interest, attitude, beliefs, values, sense of psychological safety in the learning environment, and social aspects of learning with peers and teachers.

The degree to which engagement in formative assessments affects the association between formative and summative assessments in practicing health care professionals remains underexplored. In this retrospective analysis, we examined the relationship between engagement in the ABFM Continuous Knowledge Self-Assessment (CKSA), a popular longitudinal online low-stakes formative assessment, and performance on ABFM summative assessments.

The CKSA ¹² is a low-stakes, longitudinal assessment that presents 25 online multiple-choice questions each quarter. Participants rate their confidence in each answer on a six-point scale and receive immediate feedback, including answer accuracy, in-depth critiques, and pertinent references. The assessment has no time limit, and external resource use is allowed. More than 30,000 physicians complete a CKSA each quarter. In this study, we used data from Diplomate completions of CKSA modules from 2017 to 2023.

The primary dependent variable was each Diplomate’s score on their first attempt at the high-stakes (summative) ABFM knowledge assessment from 2018 to 2023. The score was derived from either the periodic, 1-day, secure ABFM knowledge assessment (Family Medicine Certification Examination [FMCE]) or the newer quarterly Family Medicine Continuous Longitudinal Assessment (FMCLA) ¹² in use since 2019. To quantify performance of each Diplomate relative to their peers and minimize confounding from Diplomates’ ability to choose the type of summative assessment, we standardized each test score as a z score based on the distribution of scores for a given year and type of assessment (FMCE vs FMCLA). We used the continuous z score distribution for all Diplomates completing a summative exam in the study period to create three terciles of current performance (low, medium, and high). A secondary dependent variable was the score on CKSA itself because we hypothesized that some CKSA behaviors may influence the CKSA score while having no effect (or even the opposite effect) when the Diplomate eventually took the summative assessment.

We selected four measures of behavioral engagement and three measures of cognitive engagement as independent variables in our analysis. The first behavioral variable included was amount of participation, defined as the number of CKSA quarters (0–4) completed in the year prior to FMCE or FMCLA. Timing of engagement, the second variable, included the percentage of questions completed during late hours (10 PM–6 AM the next day) or on weekends (10 PM Friday–6 AM Monday). The third variable, expediency of completion (a proxy of prioritizing participation, as opposed to waiting until the last minute to complete the activity), was measured as the percentage of questions completed within the first month of the CKSA quarter. The fourth variable, concentrated participation (as opposed to distributing questions across the CKSA quarter), was assessed by dividing Diplomates into those more versus less likely than average to complete all quarterly questions in a single session.

Cognitive engagement was assessed by median time spent on questions (stratified by confidence) and median time spent reviewing critiques (stratified by correct vs incorrect answers). Most often, Diplomates answered questions in less than 5 minutes. We considered a calculated answer time greater than 10 minutes as a break in engagement; those completion times were recorded as “unknown” and were not used in calculating the median time. After looking at the frequency distributions, we excluded critique review times greater than 180 seconds because we assumed this reflected a Diplomate taking a break or interspersing CKSA with another activity.

We collapsed the six-point CKSA confidence scale into three categories: “extremely confident” and “very confident” as “high confidence,” “pretty confident” and “moderately confident” as a reference category, and “not at all confident” and “slightly confident” as “low confidence.” Less than 1% of Diplomates had missing values for high, medium, or low confidence median answer time. We replaced these missing values with the mean value among all Diplomates so as not to bias the regression results.

We included self-assessment accuracy as a proxy for metacognition. We calculated accuracy identifying strengths as the percentage of times that the Diplomate reported high confidence minus the percentage of times they reported low confidence in correctly answered questions. We calculated accuracy identifying weaknesses as the percentage of low confidence minus the percentage of high confidence on incorrectly answered questions.

Participant ability, as measured by performance on previous knowledge assessments, is associated with engagement in learning ²¹ and with performance on summative knowledge assessments. ²², ²³ We therefore included performance on the most recent prior ABFM summative examination (either certification or recertification examination) as a covariate. We controlled for gender, degree (MD/DO), active provision of patient care, location of training (domestic vs international medical graduate), and age (as a z score relative to other Diplomates being examined during a given year and with a given format [FMCE/FMCLA]). This data was derived from Diplomate recertification surveys from 2018-2023, required as part of registration for FMCE or FMCLA.

To standardize scores across years and between examination options, we transformed FMCLA and FMCE scores into z scores. We then created terciles so that we could compare Diplomates with the lowest exam performance and the best performance with the middle third. This approach allowed us to compare CKSA performance (percentage of correct questions) with examination performance.

We used χ² tests to assess the relationship between demographic variables and CKSA completion. We used t tests to examine how patterns of engagement with CKSA varied for the Diplomates in the highest versus lowest terciles of performance on the recertification exam relative to Diplomates with average performance. We performed multivariable regression to assess the effect of CKSA engagement patterns with performance on both CKSA and the subsequent summative exam. To help ensure that observed differences were related to CKSA, we controlled for performance on the previous ABFM exam and demographic characteristics. To avoid problems with circular logic, variables contingent on whether a particular CKSA question was correct (ie, time reviewing correct and incorrect answers as well as both variables relating correctness to confidence) were omitted from the regression on CKSA score.

As part of routine internal program evaluation, ethical approval was granted by the American Academy of Family Physicians Institutional Review Board.

This analysis included 24,926 Diplomates who completed a recertification exam between 2018 and 2023 (Table 1). Of these, 6,275 (25.2%) completed at least two quarters of CKSA and were therefore included in subsequent analyses of CKSA metrics.

The overall cohort summative assessment pass rate during the study period was 90.3%. Figure 1 shows the results of a primary regression examining summative assessment performance (as a z score relative to other Diplomates who took that specific exam) as a function of CKSA completion. Regression coefficients are reported for all Diplomates and three separate regressions by tercile of performance on the prior summative assessment. After controlling for demographic variables as well as prior FMCLA/FMCE performance, we found a statistically significant association between completing three out of four (P<.001) or four out of four (P<.001) CKSA quarters relative to not engaging with CKSA. Highly statistically significant associations between exam performance and completing four out of four CKSA quarters persisted for all three prior performance tercile subgroups. Additionally, we found a clear benefit to completing three out of four quarters for those in the middle tercile (P<.01).

Of the Diplomates scoring in the highest tercile of exam performance, 29.1% completed at least two quarters of CKSA vs 21.3% in the lowest tercile (Table 2). Compared with the middle tercile, those scoring in the lowest tercile were more likely to have answered questions between 10 PM and 6 AM (P<.001), spent more time per question on questions with medium or high self-reported confidence (P<.001 for both), spent less time reviewing incorrect questions (P<.001), and self-reported confidence less accurately on questions they answered correctly (P<.001). Compared with those in the middle and lower terciles, those in the highest tercile completed a higher percentage of questions during the first month of the cycle (P<.001), spent less time on questions with medium or high self-reported confidence (P<.001 for both), spent more time reviewing both correct and incorrect questions (P<.01 and P<.001, respectively), and were more accurate self-reporting confidence on correct questions (P<.001). We found no differences in answering questions on weekends, concentrated participation (answering all questions in one session instead of across the quarter), or in accuracy identifying weaknesses.

Figure 2 shows the coefficients for two separate regressions that consider the relationship of (formative) CKSA metrics with CKSA performance and later performance on the (summative) recertification exam. Prior exam score is a strong and statistically significant predictor of performance on both formative and summative assessments (P<.001 for both). Even after controlling for demographic confounders and prior score, we found an association between completing all four quarters of CKSA and higher performance on the formative and summative assessments relative to those who completed only two out of four quarters. For those completing three out of four quarters, we found a statistically significant positive effect on CKSA score (P<.001), but no corresponding effect on the later summative assessment.

Completing CKSA questions late at night or in the early morning (between 10 PM and 6 AM) had a statistically significant negative effect on the CKSA score itself but did not reach statistical significance for a deleterious effect on recertification exam performance. We found no apparent harm/benefit to completing a higher percentage of questions on the weekend (vs a weekday) or concentrating CKSA completion into a lower number of sessions. However, we found a strong association between consistently completing CKSA questions during the first month of a quarterly cycle and summative exam performance (P<.001).

Spending more time on questions with self-reported low confidence was positively associated with CKSA score (P<.001) and recertification exam score (P<.001). Spending more time on questions with medium confidence was associated with higher CKSA score (P<.001) but not better performance on the summative assessment. We observed an association between more review time after incorrect questions and higher summative assessment scores (P<.001) but did not see a parallel effect with questions answered correctly. After controlling for CKSA demographic variables and prior score, we found significant independent associations between accuracy self-identifying strengths/weaknesses and subsequent performance on summative assessment (P<.001).

To verify the robustness of our results among more homogenous pools of test-takers, we ran three further regressions that subdivided Diplomates by terciles of prior performance (Figure 3). The association between completing four out of four CKSA quarters and higher summative performance appeared strongest for those with worse performance on the prior exam (P<.001 for the lowest tercile, P<.01 for the middle tercile, and no statistical significance for the highest tercile). We found a particularly strong association between consistently completing CKSA during the first month of a cycle and performance for those in the highest tercile of prior performance (P<.001), though this association held for the other two subgroups as well (P<.05 for both). Regardless of prior performance, all CKSA users appear to benefit from spending more time on low-confidence questions as well as more time reviewing incorrect questions. Finally, we continued to observe the same association between higher accuracy self-identifying strengths/weaknesses and summative exam performance for all terciles (P<.001 for both metrics across all subgroups).

Consistent completion of (formative) CKSA is associated with higher performance on subsequent summative assessments. Specific types of behavioral and cognitive engagement during formative assessment may contribute to this effect.

If schedules allow, physicians should consider completing self-assessments early in a quarter. Doing so may be a proxy for enthusiasm about, or at least prioritization of, the task, which may lead to more engagement compared with rushing to complete the task by the deadline. Physicians should devote sufficient time to questions where they have low confidence in their answer. They should prioritize incorrect questions, even if it means de-emphasizing review of correct answers. They should avoid completing questions late at night or early in the morning, when fatigue (perhaps after a full day of other responsibilities) may make fully processing and reflecting on questions and remembering what is learned more difficult. This advice may be particularly important for physicians who scored in the lowest tercile on the previous summative examination.

Though further research is needed to clarify the mechanism, we observed that a higher ability self-identifying strengths and weaknesses during formative assessment correlated with better performance on subsequent summative assessment. This trend held across subgroups regardless of initial differences in knowledge/test-taking ability reflected in performance on the prior exam. This ability to identify strengths and weaknesses may represent a difficult-to-measure metacognitive skill. This skill may spur improvement by highlighting discrepancies between what participants think they know versus what they actually do know. ²⁴

The choice to complete questions on weekends or in a single day (as opposed to distributing effort across multiple days) did not appear to affect subsequent summative examination performance. All things considered, our findings appeared to suggest that physicians should do parts of CKSAs when they are relatively nonfatigued and can focus on learning, rather than trying to rush to meet deadlines or while attending to other competing demands.

Strengths of our study included a large, diverse sample of 24,926 Diplomates, enhancing the generalizability of the findings across a broad spectrum of practicing family physicians. The use of standardized z scores for performance comparison provided a robust method for evaluating the impact of CKSA engagement on summative assessment outcomes. The study’s comprehensive analysis of engagement metrics, such as timing, consistency, and confidence levels, offers detailed insights into how specific behaviors may influence exam performance. We controlled for a range of demographic and prior performance variables, minimizing several potential confounders that could impact the relationship between formative and summative assessment performance.

This study had several limitations. As a retrospective analysis, one can infer association but not causality. Participants who engage more consistently with CKSA might inherently differ from those who do not; therefore, results cannot be extrapolated to individuals who do not engage in such activities. While we examined several measures of behavioral and cognitive engagement, other factors that would influence our results may be missing. For example, we could not examine the correlation between flexibility or demanding work schedules and time of day of CKSA completion. We did not include measures of affective engagement, which would have required participant interviews or well-structured surveys. While we controlled for several demographic and performance-related variables, unmeasured confounders could still have influenced the results. Reliance on self-reported confidence levels and engagement metrics could introduce subjective bias. Participants were limited to physicians who completed training; findings may vary in undergraduate or graduate trainees. Finally, our findings are specific to ABFM and may not be generalizable to other specialties, professions, or certification processes. Further prospective studies are needed to validate these findings in other populations and to explore the mechanisms underlying the observed associations.

This study examined the relationship between behavioral and cognitive engagement in a low-stakes, longitudinal formative knowledge assessment and performance on high-stakes summative assessments among practicing physicians. Physicians could use findings from this analysis to potentially enhance their performance on summative examinations. Future work should prospectively examine the relationship of these formative assessment engagement factors in different cohorts of physicians (eg, by years in practice, practice type, or arrangement) with summative examination performance. Additional studies could then examine associations of these relationships with performance in practice. Participant interviews with qualitative analyses could elucidate factors including how competing demands and after-hours administrative work (“pajama time”) affect engagement. If the relationships hold, medical specialty boards and others involved in formative knowledge assessment could use these findings to develop an index of engagement. Such an index could cross-sectionally or longitudinally help identify individuals at risk of poorer examination performance (eg, those who have failed or minimally exceeded the passing threshold on previous summative assessments). Educators could use some of these strategies to coach their learners on optimizing formative assessment. Consistent with Standards for Continuing Certification, ABMS boards could reach out to potentially at-risk Diplomates to “provide [them] with opportunities to address performance or participation deficits prior to the loss of a certificate.” ⁸ This study further supports the value of structured, lifelong learning and self-assessment activities such as CKSA as part of a program of continuous specialty board certification.

This work was internally funded by the American Board of Family Medicine.