Methods for Studying Bias in Graduate Medical Education

The importance of studying bias in graduate medical education cannot be overstated. Consistent with patterns seen in the larger population, bias (ie, when one’s “preconceived opinion or judgment affect[s] their view of information, people, situations, etc”¹) in medical settings has been well established. It negatively impacts outcomes, quality of care, and patient-clinician interactions.^2–4

Due to their reliance on data and medical knowledge, medical providers can have a false sense of objectivity, especially regarding their decision-making.⁵ Medical trainees are not immune to bias seen in the surrounding culture.^6,7 Understanding the role of medical bias in graduate medical education may provide additional opportunities to intervene and work toward the goal of providing and receiving equitable care and treatment. The purpose of this article is to provide a brief and introductory review of research methods for studying bias in graduate medical education, including (a) method selection and (b) considerations for overcoming barriers to research.

There are several common ways to study bias. Unlike true experimental designs, the study of bias in graduate medical education is often conducted with trainees or faculty who work within a fast-paced and naturalistic context.⁶ Studies often use retrospective data analysis and/or self-assessment of knowledge.^4,8 The educational setting may not allow researchers to randomize and manipulate independent variables in ways that enhance scientific rigor. Given these limitations, there are several strategies to consider when studying bias or inequity in graduate medical education (Table 1).⁹

Descriptive research assists with understanding the characteristics of a population or environment. For example, how often do trainees experience bias themselves, and what types of bias exist in medical education? Descriptive research might utilize electronic or written self-report surveys, examination of medical treatments, and/or outcomes of patients treated by those working in graduate medical education.

Correlational research uses similar methodology, however analysis examines trends and/or patterns that may exist (but does not determine causality). This might include examining whether one group of trainees differs from another in their experiences, or their own patterns of treating patients. Quasi-experimental research designs use naturally occurring groups to examine cause-effect relationships, and might allow for comparisons of different cohorts, or types of residency or training programs. Finally, experimental research can be used to explore what happens when an independent variable is manipulated. Although it may be more difficult to explore and manipulate the construct of bias in medical education, use of narrative-case vignettes and/or simulated situations allow the researcher to randomize participants and directly manipulate variables of interest. 

Several self-assessment methods that are commonly used to research phenomena in graduate medical education should be applied with caution due to their flaws.⁸ Researchers often use the Implicit Apperception Test (IAT)¹⁰ to measure unconscious bias, a tool that has a variety of strengths and limitations.¹¹ Others may use self-assessment measures that are proxy or reactionary. These measures assess comfort with or knowledge of topics immediately following intervention and therefore may be limited in their accuracy. To avoid these problems, consider the use of valid and reliable measures of interest that are administered at appropriate time intervals. Valid and reliable tools provide standardized methods of assessment, reduce risk of measurement error and allow for comparisons across other studies. Being able to adequately replicate findings is a cornerstone of good quality research, and the use of reliable tools allows for more consistent and precise measurements.¹² It is preferred to use measures that have been previously validated with those working in the medical field or in educational settings, if not validated among those in graduate medical education, specifically. It is also important to set appropriate time intervals for measurement (eg, administering outcome measurement immediately postintervention does not allow for assessment of intervention or true change). Stronger designs use baseline measurements, appropriate covariates, and/or reference groups.⁸ 

The study of bias or inequity might also be examined via qualitative methods. Qualitative methods are an ideal approach when the researcher is aiming to explore and describe themes or explore a hunch prior to use of quantitative methods. These methods can be particularly helpful in understanding ways in which bias functions within an educational setting, and, at times, may be less obtrusive and more accessible than survey or experimental methods.

Qualitative research can be completed by using group interviews (ie., focus groups) or individual interviews, observational studies, or review of documents or records. For example, previous research has used thematic analysis of qualitative comments by faculty to explore gender bias in attending physicians’ feedback to residents.⁷ There are advantages and disadvantages to each method. It is important to have some training in qualitative methods or collaborate with a qualitative researcher prior to embarking on conducting interviews or focus groups, as there are a number of pitfalls with these designs.¹³

After establishing a research question and determining methodology and protocol, there may be additional difficulties to contend with. Specifically, while social desirability effects and response bias have been enduring problems in self-report research,^14,15 this is a particularly salient threat to research on bias and inequity.¹⁶ Asking culturally sensitive questions that appear face-valid may result in trainees providing socially desirable answers due to fears of retaliation and/or belief they will be perceived as discriminatory. Designing studies that ensure anonymity, providing reassurance to participants, using validated measures to control for social desirability effects, while using research assistants who do not represent a power differential (rather than having an attending physician administer the survey) are all ways to mitigate this threat.¹⁵

Conducting research on bias and inequity may provoke institutional defensiveness due to concerns that findings may reflect negatively upon an educational program. This can lead to a refusal of access to trainees, and in the worst case, retaliation against involved parties. Furthermore, should findings reflect a problematic pattern of inequity, researchers may be asked to remove data and/or be pressured to soften results.

There are several ways to protect against these threats. First, maximizing the adherence to protocol and use of the institutional review board(IRB) can be helpful. While IRBs are instrumental in the protection of human subjects, they can also serve as advocates for the researcher to uphold proper protocols. Second, researchers should aim to receive permission prior to recruiting and enlist the support of organizational leaders, even if not required by the IRB. Third, to avoid social desirability effects as well as any priming effects, it is prudent to avoid words like “bias” or “inequity” when recruiting participants. Considering more benign words like “attitudes” may avoid alarming others unnecessarily. Finally, avoid presenting preliminary data, and be mindful of requests for internal presentations to the educational organization. Others working in graduate medical education who are highly educated, but not well-versed in research, may overinterpret the tentative nature of the findings or misconstrue information as performance or program evaluation data.

Finally, as with many studies, it is wise to ensure that studies are streamlined, efficient, and as convenient as possible. Given that medical trainees and attending faculty are often inundated with other surveys, projects, and clinical duties, if a study takes a significant amount of time or is difficult to complete, it may impair sample size or result in high levels of attrition. Use of abbreviated and validated measures, research sessions scheduled during built-in didactics time or group meetings, paired with reasonable incentives to participate may result in greater participation.

Graduate medical education marks a period of substantial influence that impacts not only graduate learners, but also the patients they serve. There is a need for more research on bias and inequity that is tailored to the practical realities of graduate medical education. Areas of priority for future investigation might include (a) the role of medical faculty in perpetuating (or reducing) bias and inequity within their programs, (b) examination of the longer-term impact of experiences of inequity on graduating physicians, and (c) the downstream effects on care delivery and patient outcomes.

While there are a variety of ways in which bias and inequity can be studied in medical settings, the unique limitations of graduate medical education may pose numerous challenges that researchers need to overcome. With practical guidance, such research projects can be incorporated into graduate medical education programs in unobtrusive ways. With the rapidly changing political and cultural landscape,¹⁷ adhering to evidence-based, rigorous research on bias and inequity may be more important than ever.