Introduction: Identifying and training students who choose family medicine careers is essential to meeting primary care workforce needs in the United States. Medical students’ positive attitudes toward family medicine are associated with students’ choice of family medicine as a specialty. This study sought to refine a previously tested questionnaire assessing US medical students’ attitudes toward family medicine by shortening the questionnaire to make it more useful in educational practice and research settings.
Methods: We refined our existing 14-item questionnaire by item analysis and validation. We conducted item analysis using a graded response model approach after identifying the unidimensionality of the original scale. We selected items based on their item discrimination parameters and item information levels, and calculated the correlation between specialty choice and family medicine attitudes score to evaluate criterion validity.
Results: Exploratory factor analyses indicated the questionnaire is unidimensional. Among the original 14 items, 10 items had high item discrimination parameters and low standard error of measurement. These 10 items contribute the most to distinguishing individuals’ differences in family medicine attitudes and were selected for inclusion in the short-form questionnaire (FMAQ-S). The point-biserial correlation between the short-form scale and students’ choice of family medicine was 0.378, which provides supporting evidence for criterion validity.
Conclusion: The FMAQ-S is a concise and validated measure for assessing medical student attitudes toward family medicine. This abbreviated questionnaire can be used by medical educators to identify students for specific programming or interventions intended to support family medicine specialty choice.
In 2018, eight family medicine organizations pledged to work toward a goal: 25% of US medical school graduates specializing in family medicine by 2030.1 Identifying students with positive attitudes toward family medicine is important for reaching this goal, since positive attitudes are associated with family medicine choice2 and measuring attitude changes helps identify the impact of educational programs. Previously, we developed and validated the family medicine attitudes questionnaire (FMAQ).2,3 A shorter version of this questionnaire, with similar psychometric accuracy, may be more useful.4
We used the Item Response Theory (IRT) Graded Response Model (GRM)5 to create a short-form version (FMAQ-S) of the FMAQ. We did not approach this study with a specific number of items in mind, but rather used an empiric process to shorten the instrument as much as was feasible, without deteriorating its quality. The original 14-item instrument assessed attitudes across multiple topical domains: importance, relationships, competence and expertise, research, lifestyle, and shortage.2,3 In this analysis, favoring brevity, we approached attitude as a unidimensional construct and selected items with sound item parameters without seeking to balance the number of items by domain.
We reanalyzed data collected in 2017 from 1,188 fourth-year medical students attending 16 US medical schools. Complete details of the original data collection and student demographics were published previously.2 This study was exempt from review by the Michigan State University Institutional Review Board.
We divided collected data into two groups: developmental and validation samples (594 students each). We used the developmental sample to identify items from the original FMAQ2 that best differentiated attitudes toward family medicine. The validation sample was used to generate evidence supporting the validity of the new FMAQ-S. We used 𝟀2 analyses to compare the two samples’ self-reported demographic characteristics (race/ethnicity, gender, age, income of family of origin, anticipated educational debt, hometown size).
Item Analysis and Validation
IRT assumes all items within an instrument measure one single latent trait.6 Here, the latent trait (θ) is attitude toward family medicine. The GRM typically has two types of item parameters: discrimination (denoted as a) and difficulty (denoted as b) parameters. The item discrimination parameter indicates the degree to which a specific item differentiates individuals with different levels of a latent trait.7 The item difficulty (location) parameter (bi) represents the boundary on the continuum of a latent trait for each response category i. Hence, for an item with five response categories (eg, from strongly disagree to strongly agree), there are four category boundaries and four corresponding difficulty parameters.
We first assessed dimensionality of family medicine attitudes using exploratory factor analysis (EFA), then conducted item analyses to identify items that more effectively distinguished attitudes and had higher levels of precision. Items’ discrimination parameters (ai for item i) and item information functions (Ii[θ]) were used as selection criteria. We selected items with higher discrimination parameters, as a larger a indicates that an item better distinguishes attitude differences.8 We then assessed the item information function, choosing items that provided more information (precision) across the latent trait continuum (attitude). We selected items that provided the most information across the latent trait continuum to construct a questionnaire that captured a range of attitudes toward family medicine, rather than one with a maximum measurement accuracy at one specific cut score. We then compared the new and original questionnaires’ test information functions. To assess the new questionnaire’s validity, we analyzed the point-biserial correlation between average questionnaire scores and medical students’ intention to specialize in family medicine for both versions.
There were no statistically significant demographic differences between the two sample groups.
The Kaiser-Meyer-Olkin (KMO) test value for the original FMAQ was 0.83; values close to 1 indicate the interrelationship between items is adequate for EFA.8 The eigenvalues also supported the unidimensionality of the original FMAQ (Figure 1). The first factor’s eigenvalue was 3.50; the others were <1. The scree plot shows eigenvalues dropping steeply after one factor, meaning the first factor explained about 49% of variance. The initial FMAQ can therefore be considered unidimensional.9
Item Analysis for the FMAQ-S
Table 1 shows item analysis results using a GRM approach. Each item uses a 5-point scale, with responses ranging from “strongly disagree” to “strongly agree.” Thus, each item has four location parameters (b1, b2, b3, and b4) and a slope (a) parameter that indicate how well the item discriminates attitudes. Location b1 is the point where the probability of selecting “strongly disagree” or higher is 50%. Similarly, b2, b3, and b4 represent the locations where the probability to respond “disagree” or higher, “unsure” or higher, and “agree” or higher, respectively, are 50%.
The discrimination parameters for the 14 original FMAQ items ranged between 0.19 and 1.84. Items 2, 3, 6, and 12 had lower parameters (0.19, 0.96, 0.99, and 0.74, respectively) than other items, and provided less item information, (I<.3) across the latent trait’s points (Table 1, Figure 2). Thus, these four items were excluded from the FMAQ-S and the remaining 10 items were advanced to the validation phase.
Validation of Selected Items
Descriptive statistics and topical domains of the remaining 10 items are presented in Table 2. On the 5-point scale (strongly disagree/disagree/unsure/agree/strongly agree), means ranged from 3.38–4.36 and standard deviations ranged from 0.66–1.12. Cronbach α for the abbreviated scale in this sample was .78, indicating acceptable internal consistency, comparable to the .77 Cronbach α of the original instrument.
Test Information Function
Figure 3a illustrates test information functions and standard errors of measurement for the new FMAQ-S. This questionnaire shows acceptable information and standard error of measurement levels (θ=-4~2), suggesting it can accurately assess attitudes toward family medicine. The FMAQ-S scale exhibits a similar test information function pattern as the original FMAQ (Figure 3b). A test’s information is the sum of the information function of all the items, so the total test information increases concurrently with the number of items. The reduced number of items in the FMAQ-S means that its slightly lower I values compared to the original FMAQ are acceptable.
We analyzed the correlation between the FMAQ’s mean score and students’ intentions to specialize in family medicine to assess the FMAQ-S’s criterion validity. We used point-biserial correlation since this correlation is between an approximation of a continuous-level and a binary variable. The correlation between the FMAQ-S mean score and students’ specialty choice is 0.378, similar to the correlation in the original FMAQ (0.383, P<.01), suggesting the FMAQ-S captures the relationship between scores and intentions well.
The 10-item FMAQ-S is a concise measure for assessing medical students’ attitudes toward family medicine. It has precision and criterion validity comparable to the original FMAQ and may be easier to use in research and practice. The FMAQ-S will be valuable for evaluating the impact of educational programs on students’ attitudes toward family medicine.
Nevertheless, educational researchers should be aware of the instrument’s limitations. First, the instrument was developed using fourth-year students in the United States, and has not been tested among other populations. Second, the FMAQ-S was developed among students with a wide range of attitudes towards family medicine, and is most appropriate for use with students with wide-ranging attitudes.
Presentations: This study was presented at STFM Annual Conference on Medical Student Education, January 2021 (virtual).
- Galke C, Clements DS. 25 X 2030: anticipating the impact. Ann Fam Med. 2019;17(3):277-278. doi:10.1370/afm.2402
- Phillips JP, Prunuske J, Fitzpatrick L, Mavis B. The family medicine attitudes questionnaire: a valid instrument to assess student attitudes toward family medicine. Fam Med. 2019;51(6):493-501. doi:10.22454/FamMed.2019.957692
- Phillips J, Prunuske J, Fitzpatrick L, Mavis B. Initial development and validation of a family journal of educational statistics and PubMed medicine attitudes questionnaire. Fam Med. 2018;50(1):47-51. doi:10.22454/FamMed.2018.293942
- Weirich S, Hecht M, Penk C, Roppelt A, Böhme K. Item position effects are moderated by changes in test-taking effort. Appl Psychol Meas. 2017;41(2):115-129. doi:10.1177/0146621616676791
- Samejima F. Graded response model. In: van der Linden WJ, Hambleton RK, eds. Handbook of modern item response theory. New York, NY: Springer; 1997:85-100. doi:10.1007/978-1-4757-2691-6_5
- De Ayala RJ, ed. The theory and practice of item response theory. New York, NY: Guilford Publications; 2013.
- Baker FB, Kim S-H. Item Response Theory: Parameter Estimation Techniques. 2nd ed. Boca Raton, FL: CRC Press; 2004. doi:10.1201/9781482276725
- Embretson SE. The new rules of measurement. Psychol Assess. 1996;8(4):341-349. doi:10.1037/1040-35126.96.36.1991
- Reckase MD. Unifactor latent trait models applied to multifactor tests: results and implications. J Educ Stat. 1979;4(3):207-230. doi:10.3102/10769986004003207