Primary Care Residency Perspectives on Medical Student Performance Evaluations

It has become increasingly important for medical education programs to demonstrate the competence of their trainees beyond standardized exams so that they deliver high-value, cost-conscious, and safe patient care when practicing independently. Like Accreditation Council for Graduate Medical Education (ACGME) milestones for residencies, the Entrustable Professional Activity (EPA) framework could provide a common language for conveying students’ abilities across the continuum of medical education. While there are still limitations to the EPA model, they show promise to fulfill the Ottawa Criteria for good assessment: validity or coherence, reproducibility or consistency, equivalence with other assessment approaches, feasibility, acceptability, and a consideration of the educational effect and/or the catalytic effect on learning.^1,2

The Medical Student Performance Evaluation (MSPE) is ranked among the most important academic factors considered by primary care residency directors in the applicant selection process, alongside United States Medical Licensing Examination (USMLE) Step 1 and 2 scores, COMLEX 1 and 2 scores, and previous board failures.^3,4 The MSPE is intended to provide residency program directors an honest and objective summary of a student’s salient experiences, attributes, and academic performance.⁵ While this is the aim, there is still much work to be done in optimizing the MSPE to facilitate the undergraduate medical education (UME)- graduate medical education (GME) transition.^6-8

To the knowledge of the authors, no mixed-methods studies exist exploring primary care residency directors’ opinions on the application, and in particular, the MSPE, and how well it serves as a vehicle to make decisions to invite students for an interview. We explored which EPA skills were valued most in candidates, as these answers may help undergraduate medical education (UME) faculty and students construct the strongest primary care application, and more broadly, inform the future format of the MSPE.

The sample included PDs of any accredited (allopathic or osteopathic) primary care residency in family medicine, internal medicine, pediatrics, or medicine-pediatrics with contact information available in the American Medical Association’s (AMA) Residency & Fellowship Database (FRIEDA).⁹ This study received approval by the Penn State University Institutional Review Board prior to recruitment (study #19263). We sent an email invitation to participate to a total of 1,566 potential participants. We sent three reminder emails were sent within the 2-week study time frame.  

We invited PDs to participate in the survey online using REDCap.¹⁰ The survey contained 5-point Likert scale questions ranging from 1 (“not important”) to 5 (“extremely important”) to assess the perceived importance of student skills (eg, note writing, oral presentations, etc) and application components (eg, USMLE scores, academic rank, etc) when considering the selection of a residency program candidate.¹¹ Open-ended questions asked about additional candidate attributes and other factors taken into consideration during applicant selection, perceived strengths and limitations of the MSPE, and other additional comments. Demographics related to the participants and their programs were also obtained.

We converted Likert-scale responses for factors important to the MSPE into binary categories (yes: very or extremely important; no: not or slightly important and important). “Important” responses were added to the “no” category due to low responses for not/slightly important.  We used SAS v.9.4¹² to derive descriptive statistics. We excluded incomplete responses from analysis.  

Qualitative analysis of open-ended responses was performed using inductive thematic analysis by authors J.P. and Z.N.^13,14 We created a codebook prior to individual coding, followed by collaborative review to reach 100% agreement on final codes and themes.¹⁵

We attained 249 responses (response rate=15.9%). Participant demographics are shown in Table 1.

In terms of student skills valued by program directors in primary care, the majority identified patient communication (83.6%) and teamwork (81.9%) as most important (Figure 1). The most valued application components were being a graduate of a US medical school in the past 3 years (73.1%), no failures on board exams (58.2%), and MSPEs (54.8%; Figure 2).

Open-ended responses (n=678) yielded four main themes related to MSPE limitations (n=193) and advantages (n=193), attributes most desired in a residency candidate (n=117), other factors considered for interview (n=104), and other additional comments (n=73). See Table 2 for representative quotes.

Program directors found the MSPE to be very limited as a discriminator for applicant selection. Themes around the validity, objectivity, lack of standardization and general nature of MSPEs were identified as the most significant limitations. 

Many PDs noted the importance of student attributes such as professionalism, accountability, and teachability when considering program candidates.

PDs also noted the importance of student activities during their medical education, such as community engagement and service, leadership activities, and medical experience.

Other application components were also valued by PDs, especially the personal statement, comments from rotations, and objective data such as Step 2 exam scores.

Among primary care PD respondents, high value was placed on the MSPE, previous board failures, and Step 2 scores as tools to aid in resident selection, in accordance to 2021 National Resident Matching Program (NRMP) data, with the notable exception of Step 1 scores.¹⁶ This difference is not unexpected, as the most recent NRMP data was released prior to the shift of Step 1 to pass-fail scoring.

While many issues with MSPEs were identified, the most commonly-noted themes were around validity, objectivity, standardization, length, and the general nature of the document. Given a desire for more standardization and objectivity, using the common language of the published Association of American Medical Colleges EPAs may prove a useful, common frame of reference. We propose that the MSPE could be improved by discussing a student’s attainment of the core skills detailed in the EPAs. Schools could report how these were measured (eg, observed structured clinical encounter vs workplace-based assessment) and if competency committees were involved in reviewing the student data to improve transparency.

For primary care specialties, particular attention could be paid to the areas of patient communication (embedded in EPA 1- Gather a History and Perform a Physical Examination and EPA 11- Obtain Informed Consent for Tests and/or Procedures), teamwork (EPA 8- Give or Receive a Patient Handover to Transition Care Responsibility and EPA 9- Collaborate as a Member of an Interprofessional Team), and critical thinking (EPA 2- Prioritize a Differential Diagnosis Following a Clinical Encounter, EPA 3- Recommend and Interpret Common Diagnostic and Screening Tests and EPA 4- Enter and Discuss Orders and Prescriptions).

Program director respondents also place a high value on prior medical experience, leadership, and community service activities, as well as the congruence of the student’s interests with the program’s mission. While these aspects are likely found in other portions of the application, highlighting their impact on the school and community may improve the MSPE’s value to residency directors. Schools might also spend more time discussing the individual qualities of the student, specifically, professionalism, accountability, and character.

Though the four major areas of the country were evenly represented, respondents largely represented family medicine and there was a low number of respondents overall, making this the greatest limitation to generalizing these results. 

Table 3 lists potential strategies for addressing the MSPE issues raised in this study. More research needs to be done to explore these strategies, as revisions to the MSPE incorporating these suggested changes may be beneficial to students and residency programs on a national level.