— LEARNER RESEARCH —

Association Between Family Medicine Residents’ Mindsets and In-Training Exam Scores

Janelle Sloychuk, MSc | Olga Szafran, MHSA | Kimberley Duerksen, MSc | Oksana Babenko, PhD

PRiMER. 2020;4:33.

Published: 11/2/2020 | DOI: 10.22454/PRiMER.2020.796230

Abstract

Introduction: In medical practice, a mastery mindset is important for engaging in lifelong learning. The objective of this study was to examine the association between family medicine residents’ scores on mindset measures and their performance on in-training examinations (ITE).

Methods: This was a secondary data analysis of a cohort of family medicine residents. Following ethics approval, residents’ ITE scores from each of the 2 years of residency were linked with residents’ responses to a mindsets survey that they had taken at the midpoint of residency training. Multiple regression analysis was used to investigate the relationship between residents’ mindset scores and their ITE scores. Of 85 residents, 46 (54%) had complete data for the three data collection points.

Results: Residents’ ITE scores in year 1 were most predictive of their ITE scores in year 2 (β=0.72; P<.001). Mastery mindset scores were negatively associated with residents’ performance on the ITE in year 2 (β=-0.29; P=.004).

Conclusion: While the observed negative relationship between residents’ mastery mindset scores and their ITE performance may be disconcerting, it is not surprising. In clinical settings, residents are individually coached by preceptors and provided with specific, actionable feedback to support their learning. With respect to formative assessments, residents likely require explicit training on how to use their assessment results (ITE scores) to support their self-directed learning. This finding has practical implications for residency programs in using ITEs as formative assessments.

Introduction

An important goal of family medicine (FM) residency training is to foster a mindset of mastery for engaging in lifelong learning. A mastery mindset is a collection of beliefs and attitudes that result in a self-directed, persistent, and intrinsically motivated drive for knowledge.1,2 Such a mindset is critical for keeping up to date with medical advances and responding to changing needs of patients.

In the competency-based medical education (CBME) approach that has recently gained traction worldwide,3,4 learners are encouraged to take an active, self-directed role in their education. CBME emphasizes mastery learning and development of key competencies to improve patient care and safety.3,4 In the context of FM residency training, residents receive ongoing, specific feedback on their progress toward competence through direct observation and individual coaching by clinical preceptors,5 thus supporting the development of a mastery mindset.

Despite the global implementation of CBME, research is needed to assess impact of competency-based learning on graduating residents.6 Empirical evidence, although scarce, indicates that CBME has a potential for identifying and remediating failing learners and shortening residency training.6-8 Residents trained in an FM competency-based curriculum appear to be mastery-oriented.9 They also have lesser motivation to demonstrate competence to others for purposes of impression management (performance approach mindset) or being concerned with ego-protection (performance avoidance mindset).9 In light of this evidence and considering the mastery approach of CBME, the question arises as to whether there is an association between residents’ mindsets and their performance during residency.

Thus, the objective of this study was to investigate the relationship between FM residents’ mindsets and their performance on the American Board of Family Medicine In-training Examination (ITE).10 The ITE is a timed, computer-based assessment (240 multiple-choice questions) with good predictive validity.11 The ITE assesses residents’ knowledge and skills needed to be a family physician and reports on areas requiring improvement. At the time of the study, our 2-year competency-based residency program used the ITE as a formative assessment of strengths and gaps in residents’ knowledge and clinical reasoning to help them prepare for certification examinations.12 Although no or small positive associations between mastery mindset scores and scores on summative assessments have been reported in studies with university and high-school student populations,13 we expected a positive relationship between residents’ mastery mindset scores and their scores on the ITE, given its use as a formative assessment in our residency program.

Methods

This was a secondary data analysis of an FM resident cohort (N=85) at the University of Alberta, Canada. Figure 1 illustrates the study design. We collected data as part of two independent studies.9,12 The ITE data came from the resident cohort in year 1, and then in year 2 as part of one study.12 The mindsets data came from the same cohort of residents at the midpoint of residency training (before starting year 2) as part of another study.9 In that study, the Baranik and colleagues’ instrument14 was used to measure three types of mindsets in the resident cohort. Sample mindset items were: “I am willing to take on challenging work assignments that I can learn from” (mastery mindset; 4 items; α=0.72); “I prefer to work on tasks where I can prove my competence to others” (performance approach mindset; 4 items; α=0.82); and “I prefer to avoid situations at work where I might perform poorly” (performance avoidance mindset; 4 items; α=0.83).9 Higher average mindset scores were indicative of endorsing a respective mindset to a greater extent.

Following institutional ethics approval, we linked residents’ ITE scores in both years with their mindsets scores. Of 85 residents, 46 (54%) had complete data for the three data collection points, of whom 25 (54%) were females. Using SPSS 26.0, we performed descriptive and multivariate regression analyses. In the regression analysis, we entered residents’ ITE scores in year 1 and the three mindsets scores as independent (predictor) variables and residents’ ITE scores in year 2 as the dependent (outcome) variable.

Results

Descriptive statistics and regression analysis results are shown in Table 1. Of the four independent variables, the largest predictor of residents’ ITE performance in year 2 was their ITE performance in year 1. Mastery mindset had a significant negative effect on ITE performance in year 2. Performance mindsets had no significant effects on residents’ ITE performance in year 2. Together the four independent variables explained 66.1% of the total variance in residents’ ITE scores in year 2.

Discussion

The key study finding is that, although FM residents score high on a mastery-oriented psychological test,9 the mastery mindset appears to be negatively associated with their ITE performance. This observed negative relationship may be disconcerting at first sight; however, it is not surprising. In our residency program, which transitioned to CBME in 2009,5 clinical preceptors directly observe residents in their performance of specific competencies and provide actionable, improvement-oriented feedback to residents on a daily basis, thus supporting the development of the mastery mindset. However, the mastery mindset does not appear to transfer readily to residents’ ITE performance. We speculate that, although our residents were encouraged to use their ITE results in preparing for certification examinations,12 residents likely would require explicit training on how to interpret ITE results and use them to develop an educational plan for self-directed learning.

It is also plausible that, despite the ITE being used as a formative assessment in our program, residents may perceive it as another hoop to jump through, rather than supporting their competency acquisition needs and long-term professional growth. Interviews or focus groups with residents are warranted to test these speculations and gain a deeper understanding of residents’ perceptions of the ITE and its practical implications for residency training to support residents’ mastery learning. 

Past research has also shown that mindsets are task and domain specific.1 That is, one might possess a growth or mastery mindset in one domain or with respect to a particular task and a fixed or performance mindset in another domain or with respect to another task. The mindset questionnaire used in the study was general in nature in the context of workplace, whereas the ITE was task-oriented. As such, it is possible that residents may apply different mindsets to different tasks in different settings. Further qualitative studies testing this speculation are warranted.

With respect to study limitations, complete data had to be available for the three time points considered in the study (ie, ITE year 1 scores, mindsets data at the midpoint of residency, and ITE year 2 scores). Given that only 54% of residents in the resident cohort had complete data, it is unknown whether the observed relationships among the study variables hold for the entire resident cohort. Additionally, only one resident cohort was considered in this study due to the data availability. As such, we were unable to examine possible cohort effects. A longitudinal focused study is needed. A larger study of residents from different programs would also enable analysis by resident age, gender, and location of undergraduate medical education.

Acknowledgments

Financial support: This work was supported by funding from the Department of Family Medicine Research Program at the University of Alberta, Canada, and the Social Sciences and Humanities Research Council (SSHRC) of Canada (grant #430-2016-00267).

Presentation: This study was presented in part at the 2019 North American Primary Care Research Group (NAPCRG), Toronto, Canada.

References

  1. Dweck CS. Mindset: The new psychology of success. New York: Ballantine Books; 2006.
  2. Sproule R, Drewery D, Judene Pretti T. Development of a rubric to access lifelong learning in work-integrated learning reflection assignments. Collected Essays on Learning and Teaching. 2019;12:94-105. doi:10.22329/celt.v12i0.5376
  3. Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: theory to practice. Med Teach. 2010;32(8):638-645. doi:10.3109/0142159X.2010.501190
  4. Frank JR, Snell L, Englander R, Holmboe ES, Collaborators ICBME; ICBME Collaborators. Implementing competency-based medical education: moving forward. Med Teach. 2017;39(6):568-573. doi:10.1080/0142159X.2017.1315069
  5. Ross S, Poth CN, Donoff M, et al. Competency-based achievement system: using formative feedback to teach and assess family medicine residents’ skills. Can Fam Physician. 2011;57(9):e323-e330.
  6. Ellaway RH, Palacios Mackay M, Lee S, et al. The impact of a national competency-based medical education initiative in family medicine. Acad Med. 2018;93(12):1850-1857. doi:10.1097/ACM.0000000000002387
  7. Ross S, Binczyk NM, Hamza DM, et al. Association of a competency-based assessment system with identification of and support for medical residents in difficulty. JAMA Netw Open. 2018;1(7):e184581. doi:10.1001/jamanetworkopen.2018.4581
  8. Binczyk NM, Babenko O, Schipper S, Ross S. Unexpected result of competency-based medical education: 9-year application trends to enhanced skills programs by family medicine residents at a single institution in Canada. Educ Prim Care. 2019;30(3):152-157. doi:10.1080/14739879.2019.1573108
  9. Babenko O, Szafran O, Koppula S, Au L. Motivations for learning of family medicine residents trained in competency-based education. Educ Prim Care. 2018;29(2):86-93. doi:10.1080/14739879.2017.1362666
  10. In-Training Examination. American Board of Family Medicine. ABFM® ITE. https://www.theabfm.org/become-certified/acgme-program/in-training-examination. Accessed March 6, 2020.
  11. O’Neill TR, Li Z, Peabody MR, Lybarger M, Royal K, Puffer JC. The predictive validity of the ABFM’s In-Training Examination. Fam Med. 2015;47(5):349-356.
  12. Babenko O, Campbell-Scherer D, Schipper S, et al. Examining accuracy of self-assessment of In-Training Examination performance in a context of guided self-assessment. Fam Med. 2017;49(6):451-455.
  13. Dweck C, Molden D. Mindsets: their impact on competence motivation and acquisition. In: Elliot AJ, Dweck CS, Yeager DS, eds. Handbook of competence and motivation: Theory and application. New York, NY: Guilford Press; 2017:135-154.
  14. Baranik LE, Barron KE, Finney SJ. Measuring goal orientation in a work domain: construct validity evidence for the 2 x 2 framework. Educ Psychol Meas. 2007;67(4):697-718. doi:10.1177/0013164406292090

Lead Author

Janelle Sloychuk, MSc

Affiliations: Department of Family Medicine, University of Alberta, Edmonton, Alberta, Canada

Co-Authors

Olga Szafran, MHSA - Department of Family Medicine, University of Alberta, Edmonton, Alberta, Canada

Kimberley Duerksen, MSc - Department of Family Medicine, University of Alberta, Edmonton, Alberta, Canada

Oksana Babenko, PhD - Department of Family Medicine, University of Alberta, Edmonton, Alberta, Canada

Corresponding Author

Oksana Babenko, PhD

Correspondence: Department of Family Medicine, 6-10 University Terrace, University of Alberta, Edmonton, Alberta, Canada T6G 2T4. 1-780-248-1729. Fax: 1-780-492-2593.

Email: oksana.babenko@ualberta.ca

Fetching other articles...

Mendeley

Loading metrics from Mendeley...

Loading the comment form...

Submitting your comment...

There are no comments for this article.

Downloads & Info

Share