Maintenance of Certification Part IV in Residency

Board certification is the medical profession’s contract with society to assure physician and surgeon competence.1,2 Historically, specialties bestowed career-long board certification shortly after residency training based on performance on a single written or oral examination. Competence throughout a medical career was then generally assumed and rarely questioned. Over time medical specialties have addressed these perceived weaknesses of board certification by transitioning to time-limited certificates and adding regular practice performance evaluation and improvement to the process. In 2000, the American Board of Family Medicine (ABFM) developed a four-part maintenance of certification (MOC) process based on 3-year cycles with knowledge self-assessments (part II), practice improvement activities (Part IV), and a traditional board-style examination every tenth year.

Although in flux, Part IV activities for family physicians have traditionally included ABFM-developed online modules, modular activities developed by the AAFP called METRIC, self-directed quality improvement activities approved by the ABFM, and other previously approved external quality improvement activities.3,4 Additionally, ABFM participates in the American Board of Medical Specialties (ABMS) Multispecialty MOC Portfolio Program, which approves external Portfolio sponsors to then review and approve quality improvement activities for Part IV credit.1,5

MOC for practicing physicians has been controversial, with the Part IV requirement being particularly concerning for many physicians.6-9 Some report MOC Part IV activities as beneficial to patient care.10-14 The ABFM also reports general satisfaction with online modules for Part IV completion.15 However, a focus group of board-certified internal medicine and family medicine physicians identified MOC as overly complex, with Part IV activities being particularly difficult and providing limited value.8 Some family physicians have expressed concerns about broad MOC mandates in light of the limited evidence base for its inclusion in board certification.16,17

Despite some controversy with MOC in general, the ABFM has now extended participation in the MOC process into residency. Starting in 2012 the ABFM requires all residents sitting for their initial certification examination to complete one part II activity, one Part IV activity with patient-level data, and one other activity of their choosing.18,19 Although it aligns with Accreditation Council for Graduate Medical Education (ACGME) requirements for quality improvement work during training, resident participation in MOC was novel and based on little published evidence specific to doing so in the graduate medical education setting. One survey of family medicine residency directors reported 39.3% of programs used Part IV activities prior to the ABFM requirements, presumably for specific educational or assessment purposes.20 The same survey suggested a sizeable majority of residency directors planned on using either ABFM or American Academy of Family Physicians-(AAFP) produced modules to satisfy the then soon-to-be implemented requirement. Little is reported on the current use of various Part IV activities by family medicine residents since inception of the requirement, and resident educational and patient care outcomes pertinent to completing Part IV activities by residents is unknown.

Requiring Part IV activities during residency may produce the same concerns voiced by practicing physicians such as lack of comfort or skills, poor relevance to practice, and creating an undue burden. Applying the process to an academic setting may also raise new concerns such as recruiting qualified supervising faculty.21 Integration of the new requirement into an established curriculum could raise duty hour and scheduling questions.

The effect of requiring Part IV activities on all family medicine residents is unknown and presently unstudied. This study examined the perceived relevance, usability, and effectiveness of these alternatives by residents of our four affiliated family medicine programs in three common options for Part IV: (1) ABFM-approved self-directed project; (2) online module; and (3) ABMS Multispecialty MOC Portfolio Program. Our research questions were: (1) What are the participants’ perceived benefits and barriers to three methods of Part IV? and (2) What are the participants’ perceived impacts on practice of each method? Through an initial survey followed by focus groups of family medicine residents, we studied perceptions of the Part IV experience and preferences in the three methods used in a graduate medical education setting.

The four family medicine residency programs affiliated with the University of Kansas School of Medicine served as the study sites. Investigators invited all current residents who had previously completed a Part IV activity to participate in the study. At the time of the study, residents utilized either self-directed projects, online modules by the ABFM or the AAFP (METRIC), or Portfolio-approved activities. Residents selected their method to complete the Part IV requirement independent of this study and based on internal residency program factors. Residents completed all work for the Part IV activities prior to contact by the study investigators. Potential participants at the time of the study included 28 residents at the two Wichita campuses, seven residents at the Kansas City campus, and 13 residents at the Salina campus. Investigators informed all potential participants of the study by email and contacted as many as possible through in-person visits to the programs’ typical meetings. All study participants gave consent per institutional review board (IRB) protocol. Out of a potential study population of 48 residents, 26 residents participated in the study (54% response rate). Approval for this study was granted by the IRB at the University of Kansas School of Medicine-Wichita.

Residents reported a wide variety of clinical topics as the focus of their quality improvement work in completing the requirement (Table 1). Twelve residents reported using ABFM or AAFP METRIC modules, eight residents reported self-directed quality improvement activities, and six residents reported using a Portfolio-approved activity to fulfill the Part IV requirement.

After agreeing to participate, 26 residents completed a 10-minute written survey exploring perceptions of the completed Part IV activity. The survey queried ease of use, convenience, relevance and impact on practice, and benefits and barriers to participation. Upon completion of the written survey, the 26 participants were broken into investigator-led focus groups of three to six residents to elicit more details regarding their experience. Study personnel used a semistructured question guide to facilitate the 30- to 60-minute group discussions. Over a span of 7 months, one focus group occurred at each of the four residency sites. Investigators conducted one additional resident focus group at one of the sites due to a large number of residents finishing self-directed activities at the end of the project timeline. Ultimately, the study included five focus groups with all sessions being audio-recorded and coded for themes with independent analysis by three researchers. No participant names were recorded or used in the analysis.

The written survey included Likert scale response questions (5-point scale from strongly agree to strongly disagree), pick list questions (“check all that apply”), and open-ended free text questions. Descriptive statistics were used to identify topics and activity chosen, and analysis of variance (ANOVA) was used to compare differences between resident perceptions of the type of Part IV activity chosen. All data analyses were performed using SPSS version 24.0 (IBM, Armonk, NY). Two investigators (ND, KN) also categorized open-ended free text survey responses for themes using a phenomenological approach.22 These categorized themes were then stratified by Part IV activity completed and analyzed for differences.

Recorded focus group data supplemented the findings of the written survey and concentrated on two specific areas: concerns about Part IV, and the impact on practice of completing the Part IV activity. One investigator (ND) analyzed the comments for common themes to create a framework of categories against which to analyze the data. All three investigators then independently analyzed comments against the framework and compared for final classification with interinvestigator mismatches adjudicated by consensus. Another investigator (KN) separated the focus group comments by Part IV modality used by the resident and themed comments were categorized by Part IV activity completed.

Twenty-six residents completed most or all of the questions on the written survey. Residents averaged 30.6 years of age (range 27-42). Twelve residents (46%) were female. All residents reported less than 5 years of practice (Table 2). The majority of residents reported working on Part IV activities in a clinic setting (17/26), with fewer doing activities at home (5/26) or the hospital (4/26). The majority of residents reported completing the activities either outside of clinical time (9/26) or with a combination of both clinical and outside clinical time (16/26). Only one resident reported completion of Part IV activities strictly during clinical time.

Time was the most consistently identified barrier or challenge to participation across all types of Part IV activities in a “check all that apply” question. It was the most often selected barrier for residents who completed online modules (10/12) and self-directed activities (7/8), and was the second most identified barrier for the portfolio projects (3/6). Residents doing portfolio projects identified lack of expertise in data analysis as the most common barrier (6/6), while this was the second most selected barrier for online modules (2/12) and self-directed projects (4/8). In an open, free text response to the “biggest concern regarding MOC Part IV requirements,” residents most commonly identified lack of time to be of primary concern (9/18 comments; Figure 1).

In a “check all that apply” question, residents in portfolio projects most commonly identified improved patient care as a benefit of participation (5/6), while this was the second most commonly selected benefit for self-directed activities (7/8) and third most commonly selected benefit for online modules (5/12). Relevance to practice was the most commonly identified benefit for online modules (7/12) and self-directed activities (8/8), while it was the second most identified benefit for the portfolio activities (3/6). Residents most commonly identified improved patient care (12/24 comments) and improved understanding of quality improvement principles (9/24 comments) in an open, free text response to “how will completing the MOC activity impact your practice?”

A one-way between subjects ANOVA was conducted to compare the responses to six 5-point Likert scale questions asked on the survey. These questions asked about the chosen MOC Part IV activity’s relevance to their practice, whether performing the activity caused them to make a change in their practice, the ease of activity, whether or not the activity improved their practice, if they would recommend the activity to colleagues, and whether or not they would participate in the activity if it were not required (Table 3).

All responding residents reported they “somewhat agree” or “strongly agree” that their activity “was relevant to my practice” with more selecting “strongly agree” for self-directed (6/8) and portfolio (5/6) projects compared to online modules (2/12). There was a statistically significant difference between the resident responses regarding the relevance of the type of activity to their practice [F(2, 20)=7.18, P=.004]. Post hoc comparisons using the Tukey honest significant difference test indicated that the mean score for the online modules (M=4.20±.42) was significantly different than portfolio (M=4.8±.38) and self-directed activities (M=4.8±.41).

Residents completing online modules found the “activity easy to accomplish” more often (9/12 “strongly agree” or “somewhat agree”) compared to self-directed activities (2/8) and portfolio projects (3/6). There was a statistically significant difference between the resident responses regarding the ease of use of the type of activity [F(2, 20)=4.61, P=.023]. Post hoc comparisons using the Tukey honest significant difference test indicated that the mean score for the online modules (M=4.20±.63) was significantly different than self-directed activities (M=2.7±.95). However, the portfolio activity (M=3.0±1.67) did not significantly differ from the online and self-directed modules in terms of ease of use.

Despite attesting to its ease of use, online module residents tended to “strongly disagree” or “somewhat disagree” to the statement “I would participate in this activity even if it were not required” (8/12) more so than for self-directed activities (2/8) and portfolio projects (2/6), although this did not reach statistical significance. While also not statistically significant, residents were more likely to select “strongly agree” or “somewhat agree” for self-directed activities (7/8) and portfolio projects (5/6) compared to online modules (6/12) for “what I learned in this activity will improve my practice.” Finally, answers were largely mixed across type of Part IV activity for “I will make a change in practice based on this activity” and “I would recommend this activity to colleagues.”

Several themes emerged during the focus groups regarding concerns about participating in Part IV activities. First, technical issues in completing the application to gain credit for the activity seemed problematic for all categories of Part IV. Residents focused on issues with the various websites or the application processes, not on the quality improvement methods themselves. The time consuming nature of MOC seemed more pronounced with the self-directed and Portfolio options compared to the modules, which were viewed as more “spoon-fed” to the participants. Residents in all categories expressed the feeling that participating in Part IV MOC activities distracted from other important activities in their limited time. Residents using the online modules in particular identified the activity as simply another “box to check.” Lack of knowledge on how to complete the projects and a lack of patients to fulfill the requirements of the activity was also a significant concern for all three categories.

When asked about perceived impact on practice, residents in the self-directed and Portfolio groups generally listed some benefit to patient care in the discussion, while residents using modules almost universally agreed that the project had no or very limited effect on patient care after the project finished. In the discussions, it emerged that ABFM and METRIC modules tended to be completed individually and self-directed projects and Portfolio projects were completed in teams, with teamwork identified as a benefit of these Part IV methods. Portfolio projects seemed to provide the highest-rated relevance to practice and educational value, followed by self-directed projects, and then ABFM and METRIC modules.

As Part IV MOC spreads from certifying the good physician2 to an activity in educating the good physician, it becomes important to evaluate the effect of this relatively new requirement on the residents of family medicine training programs. Our study of residents does suggest a possible preference for Portfolio and self-directed activities over online module activities, especially in terms of perceived relevance to practice and impact on patient care, even in light of comparative ease of use of the modules.

The reason for this possible preference is not known, although it is an interesting observation that Portfolio-approved and self-directed activities amongst our residents were all group projects while ABFM and METRIC modules were generally completed individually. Our study might suggest that the group QI performed through self-directed and Portfolio activities added perceived value to the activity compared to individually-completed module activities. Whether the preference for portfolio and self-directed activities was related to the group nature of their projects or to some other intrinsic factor of the specific methods of gaining Part IV credit is unknown.

The literature reports several group Portfolio-approved activities with teamwork identified as a valued part of the experience. One suggested that external supports, like QI coaches and data collection, added to the success of Portfolio group QI activities.8 Overcoming the lack of QI knowledge or skills and lack of time may make the Part IV activity more valuable.17,21 Participants in our study did not specifically comment on external support as adding value to their activities and we do not know if such support was available to them. They did, however, report teamwork as a significant benefit to the group QI activities.

The ABMS has identified the comparison of completion of Part IV activities individually versus as a group as worthy of exploration.9 Prior to the resident Part IV requirement, approximately 40% of family medicine residency directors reported using Part IV activities with most using ABFM- and AAFP-developed modules.20 It is unknown if they were used individually or in teams, although the same survey indicated 67.7% of residency directors planned to use the ABFM and AAFP modules when Part IV activities became a requirement with 48% planning on using the modules to guide group activities. Only 19.7% planned to use the modules for individuals. The actual use of ABFM and AAFP modules individually versus group quality improvement initiatives across family medicine residencies since that report is unknown. The effect of doing group QI using ABFM or AAFP modules has limited, but positive, evidence.23 Our results may have been different if the residents using online modules had happened to use them in a group QI setting.

Our study has several limitations. First, the study surveyed a small group of participants in four university-affiliated family medicine residency programs. Initial impressions may not be widely held across all family medicine residencies or even all family medicine residents at our four programs (ie, possible selection bias). This limitation could be overcome through a more broadly distributed survey of all residents through other mechanisms, such as CERA surveys or reports of participant feedback when Part IV activities are completed through the various options.

Second, our study evaluates participant perception, rather than actual patient care or educational outcomes. Further study on participation in Part IV activities during residency should also include evaluation of increased knowledge and skills in patient care and health systems improvement. Whether participation in Part IV activities increases success for board certification and participation in the current MOC process could also be areas of investigation. Actual patient outcomes for residency-required Part IV activities should also be examined.

Finally, MOC is in flux with new options currently available for completion of the Part IV requirement that did not exist at the time of our study. For example, the ABFM recently announced a new Part IV option called the Residency Program Performance Improvement Program (ResPIP) that is available to residency programs that have a demonstrated ability to develop and oversee their own QI projects.24 Although unstudied, ResPIP’s intent to increase relevance for resident QI projects may address many of the concerns raised in our study. The shifting landscape in MOC may limit the generalizability of our study results to the current environment.

Despite these limitations, this study begins to shed light on the effect of requiring Part IV activities during residency. Among our family medicine residency participants, Portfolio-approved and self-directed activities completed in a group setting seemed to be the most positively reported way to complete the ABFM requirement. Regardless of path chosen, time constraints and QI expertise are significant barriers to completion of the requirement. Residency programs will need to grapple with these barriers to maximize the benefit to their residents as they prepare to become board-certified family physicians.