Validation and Comparison of a Brief Instrument vs a Single-Item Screen to Predict Entry to Family Medicine at Matriculation to Medical School

The United States lacks sufficient primary care physicians to meet the needs of the public and ensure optimal health outcomes.1 Medical students who enter family medicine represent the largest contribution to the primary care workforce compared to other specialties.2 Much is known about the demographics, student factors, and medical school experiences that are associated with student choice of family medicine, including curricular and extracurricular offerings.3 Models of primary care career choice posit that although it is a complex process, initial interest in either the specialty or the type of work done by the specialty is key.4

Identifying early interest in family medicine could guide efforts to nurture student interest in and exposure to programs, activities, and training opportunities that promote entry into the discipline.5,6 However, there is no reliable method to identify whether matriculating medical students are likely to become family physicians. The European Academy of Teachers in General Practice created a 60-item questionnaire to predict attitudes toward and entry into family medicine after clerkship completion, but it does not appear to have been widely adopted.7 It is possible that the length of the instrument prevented its application; shorter questionnaires have been suggested to lessen response burden and improve response rates.8 An 83-item survey about family medicine perceptions was administered among Spanish medical students, although this instrument did not undergo validation beyond calculation of Cronbach α.9 Similarly, a 25-item survey that assess attitudes toward primary care for premedical students exists, although it also lacks formal validation.10 More recently, a 17-item instrument to assess medical student attitudes toward family medicine during medical school has been validated and appears to correlate with career choice, although this was also not administered upon matriculation.11

The first goal of this study was to validate the Family Medicine Interest Survey (FMIS), a 15-item instrument created from an existing longer unvalidated career preferences survey administered at UWSOM, to identify student interest in family medicine upon matriculation. The construct of the FMIS rests on the idea that family medicine is the broadest of medical specialties and can be defined as the care of undifferentiated patients presenting with undifferentiated problems.12 The FMIS includes items with explicit queries about student interest in primary care and family medicine. Other items cover scope of care (eg, acute, chronic, procedures) and range of practice environments (eg, rural, underserved areas).13,14 Items also reflect other characteristics of primary care (eg, continuous relationships, care coordination, etc).15 The second goal of this study was to compare use of the FMIS versus a single-item screen asking about top specialty choice to predict which matriculating students will select family medicine as a career.

Students at the University of Washington School of Medicine (UWSOM) complete an internal survey called the Biographical and Career Preferences Inventory (BCPI) upon matriculation to medical school, usually during their orientation week. The entering classes between 2003 and 2007 had 784 students; of these, 684 completed every item. Demographic variables including age, gender, and race/ethnicity were identified from students’ American Medical College Application. The 2015 American Medical Association (AMA) Masterfile was used to identify the eventual practice specialty for each student; this information was available for 601 students. The data set was created and deidentified by the UWSOM Department of Biomedical Informatics and Medical Education prior to analysis. This study was deemed exempt by the University of Washington Institutional Review Board (IRB) and was approved by Oregon Health & Science University IRB (16592).

Oregon Health & Science University (OHSU) collected students’ top three choices of specialty upon matriculation starting in 2006. This was linked to residency match data for graduates from 2010 to 2016 by the Department of Family Medicine and deidentified including all demographics prior to analysis. Of the students who graduated between 2010 and 2016, 830 completed the matriculation survey asking about their top three specialty choices, 744 had complete data for the analysis, and 147 of these students matched to family medicine residencies.

Fifteen items were extracted from the BCPI to construct the FMIS to measure family medicine interest at matriculation. The first two items ask students to rank their interest in family medicine and primary care on a 5-point Likert scale, with 1 being strongly inclined to avoid and 5 being strongly inclined to select. Students are asked to rank 13 clinical factors on a 5-point Likert scale with 1 being very unimportant and 5 being very important in their preferred specialty. For ease of calculating difficulty and discrimination analyses, these Likert scales were converted to range from 0 to 1. Thus, the maximum score of the complete FMIS is 15. The BCPI also includes a question used to construct the single-item screen: a write-in response asking students to list their current preferred medical specialty. The FMIS and single-item screen is shown in Table 1. Difficulty analysis, discrimination analysis and Cronbach α were calculated to determine test performance measures of the FMIS. 16,17

Difficulty analysis was determined by comparing the expected standard deviation of each item to the actual standard deviation seen for each item. Since all items used a 0 to 1 scale the expected mean for normally distributed responses would be 0.5 with an expected standard deviation of 0.167. Means falling outside 1.5 standard deviations (ie, means between 0.25 and 0.75) would indicate less variation in responses, suggesting that the item did not contribute much to the α value of the scale. The overall variance of responses was determined by viewing whether standard deviation of each item was more than the expected standard deviation of 0.167.

Discrimination analysis determines whether the item helps in distinguishing respondents who score high vs low. To calculate item discrimination, the total scores for each respondent are calculated, and then divided into three groups: upper, middle, and low. For this analysis, 30% of the total scores were each in the upper and low groups with 40% of total scores in the middle group. Then for each item, the mean of the lower group is subtracted from the mean of the upper group; this difference is known as D. The standard deviation of each item is then compared to D. If the standard deviation is more than D, the item may not discriminate between respondents with high total scores from low total scores. For internal consistency Cronbach’s α was calculated with all 15 items with a minimum acceptable α of 0.7.

In addition to these test performance measures, face validity of the FMIS was determined by content analysis of an expert panel of family medicine educators that were members of the Society of Teachers of Family Medicine and part of the steering committee for the Group on Medicine Student Education in a conference call in May, 2016.

Criterion validity was established using the outcome of family medicine practice for UWSOM and for family medicine residency match for OHSU data. All analyses were performed with SPSS Version 21, IBM.

For UWSOM data, the variable of writing family medicine as the top specialty choice was effect coded: the response “family medicine” given 1, all other responses -1. A composite score for the 15 items was created with a minimum score being 0 and a maximum score being 15. This score was then standardized using z scores. Practice outcomes for each of the three primary care specialties (internal medicine, pediatrics, family medicine) were dummy coded, with 1 being that particular specialty and 0 being all others. Three variables were created to control for demographic differences in specialty choice: student age, gender, and race/ethnicity. The age of the student at the time of match to residency was standardized using z scores. The gender of the student was effect coded, with 1 being female and -1 being male. Race/ethnicity was also effect coded, with 1 being a person of color and -1 being white. Binary logistic regression with standard predictor entry was used to establish the relationship between the outcome variables (family medicine, pediatrics, or internal medicine practice) with the FMIS and the single-item screen (listing family medicine as top choice) while controlling for the demographic variables of age, gender, and race/ethnicity.

For OHSU data, the outcome variable was matching to a family medicine residency, which was dummy coded, 1 being family medicine and 0 being all others. The predictor variable was writing in family medicine to the top specialty choice upon matriculation, which was also dummy coded with 1 being family medicine and 0 being all other write-in options. Binary logistic regression with standard predictor entry was used to establish the relationship between the outcome variable (family medicine residency choice) with the predictor variable of the single-item screen (listing family medicine as top choice) while controlling for the year of residency match.

Table 2 shows item and reliability analyses for the FMIS. The overall nonresponse rate ranged from 0% to 1.4%. Means for the items ranged from 0.50 to 0.89. Seven items had means above the desired 0.75 (primary care, patient education, personality fit, continuity of care, diverse patients, patient relationships, and challenging practice). The SD for items ranged from 0.18 to 0.27 and no items had a SD below 0.167 that would indicate less than desired variance for that item. Six items showed less than desirable discrimination as indicated by the SD being higher than the value for D (procedures, personality fit, acute care, clinic setting, challenging practice, and flexible practice). Cronbach α was 0.767, which is in the acceptable range.

Both the FMIS and the single-item screen were uniquely predictive of later practicing family medicine, as shown in Table 3. UWSOM graduates listing family medicine first had an odds ratio (OR) of 2.00 for practicing family medicine compared to those listing other specialities. For the FMIS score, the OR of practicing family medicine increased by 1.33 for every one standard deviation (SD) above the mean. Regression coefficients were used to calculate predicted probabilities of entering family medicine based on combinations of listing family medicine first and SD of score on the FMIS, shown in Figure 1. Students with high interest in family medicine at matriculation, as indicated by a score more than 1 SD above the mean and listing family medicine as their top specialty choice had a predicted probability of 62% of eventually practicing family medicine. Conversely, students with a score less than 1 SD below the mean who listed a specialty other than family medicine as their top choice had a predicted probability of just 9% of eventually practicing family medicine. Results based on the same set of predictors did not show that the 15-item score or listing family medicine as the top specialty choice uniquely predicted eventual pediatric practice or internal medicine practice (data not shown.)

Listing family medicine as top specialty choice was uniquely predictive of matching to a family medicine residency for OHSU graduates, as shown in Table 2. OHSU graduates who listed family medicine as their top career choice upon matriculation had an OR of 8.28 of matching to family medicine residencies compared to those who listed another specialty first.

Previously developed surveys and instruments to measure interest in family medicine have either been lengthy, lacked formal validation, or were not administered upon matriculation to medical school. The FMIS is a reliable test to determine initial interest in family medicine at matriculation. Although 7 out of 15 items of the FMIS had means outside the desired range, this is likely due to the very low number of respondents using the lowest point of the rating scale and thus it is acceptable to retain these items. This conclusion is further strengthened by the fact that all items had a standard deviation larger than desired, indicating acceptable variance of the item responses. The nonresponse rate was low for all items, despite the fact that the items used are toward the very end of the BCPI. Based on the overall Cronbach α of this scale and the lack of meaningful improvement in the α if items were deleted, it is acceptable to retain the current items and proceed with the FMIS for test interpretation.

For test interpretation, results were linked to eventual practice outcome and therefore criterion validity was used to determine whether the FMIS and the single item screen predicted who went on to eventually enter family medicine. Different criteria were used for UWSOM (practice outcome) and OHSU (residency specialty) for the single-item screen. At both institutions, listing family medicine as the top specialty choice upon matriculation was associated with higher predicted probability of entering the specialty compared to listing another specialty.

For UWSOM, combining the FMIS with the single-item screen predicted who ended up practicing family medicine with a dose-response relationship. Students who listed family medicine as their top choice and scored high on the FMIS were predicted to enter family medicine at much higher rates than those who listed another specialty and scored low on the FMIS. As indicated by the ORs, listing family medicine first was a better predictor of practicing family medicine than a higher score. This suggests that simply asking students their top specialty choice upon matriculation may be the briefest possible test to determine who is likely to become family physicians.

Strengths of this study include the large number of students represented (total sample size of 1,345) and the use of the AMA Masterfile to determine final practice outcomes for the UWSOM students. Another strength is that the findings of the single-item screen are consistent across different time periods for different medical schools, from students who matriculated to medical school between 2003 and 2007 at UWSOM, and between 2006 and 2012 at OHSU. Limitations of the study include that it took place at two public medical schools in the Pacific Northwest, so results may not generalize to private institutions or to schools in other geographic regions. Another limitation is the lack of additional variables from OHSU beyond initial specialty interest to adjust for in the regression model, which is probably the reason the OR for single-item screen for this instution was so much higher than that of UWSOM.

These shorter validated instruments are an improvement over the prior approaches and have the benefit of predicting who is potentially interested in family medicine on the very first day of medical school. Future directions for research include confirmation of the validation of the FMIS and the single-item screen at other institutions. Furthermore, this study also looks at only two periods of time, one very early in medical education (matriculation) and then others quite late (eventual practice or residency match), thus disregarding how specific experiences in medical school could influence student career decisions. Future studies should examine the association of various medical student experiences on students who enter school with different levels of family medicine interest, as scored by the FMIS and the single-item screen.

Both the FMIS and single-item screen can identify students at matriculation who are more likely to enter family medicine. Combining both tool leads to a better prediction ability compared to use of either alone. However, for schools that have limited ability to survey students upon matriculation, the single question of “What is your current preferred medical specialty?” can help family medicine educators quickly identify students who are more likely to enter the field. Schools that face institutional barriers to implementing a longer survey gauging student interest in family medicine should argue for the inclusion of the single-item screen in the orientation and matriculation activities of all students when they start medical school, as this still has good predictive value for entering family medicine. Early identification of interested students could allow family medicine departments to direct their limited resources to the students already most likely to fill the primary care pipeline.