The Science and Art of Mixed Methods in Medical Education Research

Sitting at the intersection of biological and social sciences, medical education is a fitting context for mixed methods research—research that explicitly integrates qualitative and quantitative approaches to leverage their complementary strengths and weaknesses.¹ Qualitative research explores and illustrates lived experience.² It provides depth and richness in response to research questions. Quantitative research focuses on measurement and is better for examining associations.³ Mixing the two approaches can provide a comprehensive understanding of a topic.

Combining qualitative and quantitative approaches in medical education research is also beneficial given that the field involves complex initiatives that are coordinated among multiple parties.⁴ Further, patient care often mirrors mixed methods as clinicians gather, compare, and interpret both qualitative patient histories and quantitative test results.⁵ However, formal mixed methods training is rare along the medical education pathway. Training and workshops generally reach only funded physician researchers and physician scientists.⁶ In this article, we introduce mixed methods research, describe its underlying principles, and explain key considerations involved in its conduct.

The precise origins of mixed methods research have been debated, though scholars agree that it is not new.⁷ Mixed methods have been used in the social sciences for nearly 100 years,⁸ although some scholars trace it back to ancient Greece.⁹ Other experts point to the broader uptake of mixed methods across disciplines in the 1980s, including psychology, sociology, anthropology, and education.¹⁰ In family medicine, interest in mixed methods grew because of its ability to draw on multiple tools to provide comprehensive insights, thereby advancing a field that maintains a generalist model rather than one pursuing specialization.¹¹

The ability of mixed methods research to integrate distinct approaches is rooted in the philosophical foundations of both qualitative and quantitative research. The philosophical underpinnings of science, or research paradigms,¹² include five components that inform how science is understood, conducted, and valued: ontology (researcher’s view of reality), epistemology (how the researcher thinks knowledge is created), axiology (role of values in research), methodology (approach and processes for conducting research), and rigor (criteria used to assess research quality).¹³ Table 1 outlines quantitative, qualitative, and mixed methods research paradigms across these domains.

Quantitative studies often follow a positivist paradigm,¹³ which is prevalent in medical education research. Taking a positivist view assumes that a single reality exists that can be measured and studied to produce knowledge that is neutral and free of individual values. Positivist studies involve testing hypotheses and using the scientific method to identify causality through prediction and control. For example, Dunleavy and Paladine examined the effect of student-run free clinics on the number of medical students matching into family medicine.¹⁴

In contrast, qualitative studies often follow a constructivist paradigm,¹⁵ which views reality as socially constructed and therefore assumes that multiple realities coexist and change over time (i.e., no single reality exists because how each person experiences the world is different). Researchers who take a constructivist perspective explore how people create meaning through their interactions with others. Accordingly, constructivists apply inductive approaches that elicit unique perspectives about how people understand the world around them based on their social and historical placement in society. For example, Sanders and colleagues aimed to understand faculty and resident experiences with sexual harassment, gender bias, and gender discrimination.¹⁶

In the late 1980s, philosophical debates about opposing positivist and constructivist research paradigms led to the rise of mixed methods as an alternative.¹⁷ To adequately integrate these seemingly incompatible worldviews, proponents of mixed methods research sought to define its philosophical orientation. The research paradigm most often associated with mixed methods is pragmatism, which contends that knowledge is created from the interaction between people (constructivism) and their environment (positivism). Thus, pragmatists contend that multiple realities can exist within a single physical world.¹⁸ Compared to a positivist view where the researcher believes their work is separated from their values and that they can control threats to validity, a pragmatist prioritizes participant values in pursuit of findings that are ecologically valid. Researchers who take a pragmatist approach draw on quantitative and qualitative methods, applying methods that are best suited to the research question they seek to answer. For example, to comprehensively evaluate an experiential interprofessional education program, Miselis and colleagues used quantitative surveys and qualitative interviews, as well as focus groups of trainees.¹⁹ Using both approaches allowed the researchers to reach a deeper level of understanding regarding the impact of their intervention.

Mixed methods research addresses an overarching research question through the integration of quantitative and qualitative methods in a single study. Mixed methods is not just putting quantitative and qualitative methods side by side. Through integration, researchers gain insight that they could not discover without combining the two methods. Researchers generally apply mixed methods for one of five purposes: triangulation, complementarity, development, initiation, or expansion.²⁰

When researchers apply mixed methods for the purpose of triangulation, they apply two or more methods that have inherently different assumptions and biases, increasing the validity of findings. Researchers compare the quantitative and qualitative findings to identify where findings converge or diverge. As an example, Sherbino and colleagues followed qualitative inquiry with a national survey to validate results.²¹ In their discussion, they specified that the quantitative survey findings demonstrated the absence of bias in the study.

When researchers apply mixed methods for the purpose of complementarity, mixing methods can increase the interpretability and meaning of research. Sebok-Syer and colleagues applied mixed methods to understand how electronic health record (EHR)-based metrics were used in resident assessment.²² The researchers combined quantitative data from the EHR and qualitative inquiry with resident interviews methods to enrich their understanding. By layering qualitative data over quantitative results, researchers were able to obtain more complete conclusions about resident performance.

A common purpose of mixed methods in medical education is development.²⁰ Researchers use one method to inform the other to increase the validity of constructs and measurement. To explore sexism in the surgery discipline, Barnes and colleagues conducted qualitative inquiry with focus groups to create measurements for the quantitative phase.²³ This integrated approach enabled the researchers to include measurements that were situated within the experiences of the population of interest.

A less common purpose for mixed methods is initiation. In these studies, researchers apply two distinct methods to discover divergence. Studies aim to explore paradoxes, looking for nonconvergence to more fully explain experiences. In a study with Latino day laborers, Negi discovered a discrepancy between the study’s quantitative and qualitative phases,²⁴ which highlighted the unexpected role of religiosity as a protective factor.

When researchers apply mixed methods for the purpose of expansion, mixing methods extends the scope, breadth, and range of inquiry by using different methods for different inquiry components. Expansion is a common purpose in program evaluation, where researchers use quantitative methods to assess program outcomes and qualitative inquiry to understand experiences or implementation. As an example, Cronholm and colleagues applied mixed methods to assess the impact of a learning collaborative on experiences in their training environment.²⁵

After researchers determine their purpose for applying mixed methods, they identify what design to use. Although methodologists suggest different typologies of mixed methods designs across disciplines, family medicine researchers generally follow one of three designs: convergent, sequential, or embedded.²⁶ Designs are characterized by the sequencing and weighting of the qualitative and quantitative component.¹ Quantitative and qualitative data can be collected concurrently (in parallel, at approximately the same time) or sequentially (one follows the other). In weighting, researchers give the quantitative and qualitative inquiry equal or unequal importance in the study.

In convergent design, the quantitative and qualitative components occur concurrently.¹ Researchers ensure overlap between the qualitative and quantitative goals. For example, Aagesen and colleagues applied a convergent mixed methods approach to evaluate the effectiveness of guided discovery learning in teaching surgical skills.²⁷ They used quasi-experimental data to establish efficacy of the intervention and interview, and observational data to explain group differences. Although the researchers gathered information through two different methodological approaches, both the qualitative and quantitative aspects were aimed at determining intervention effectiveness.

In convergent design, analysis is usually simultaneous and focused on identifying how each component supports (converges) or does not support (diverges) from each other. Researchers show the qualitative and quantitative findings next to each other (in text or tables) to allow for greater contextualization. For convergent studies, data can also be transformed. The qualitative or quantitative data can be reformatted into quantitative or qualitative data so that it can be merged with its respective quantitative and qualitative counterpart, allowing for further analysis.

In sequential design, one phase builds on results from another. One of two types of sequential design, exploratory or explanatory, is selected depending on the purpose of mixing.²⁶ In an exploratory sequential design, the qualitative phase shapes the quantitative phase. Exploratory sequential design is commonly used to develop or adapt survey instruments or scales that can be used to gather more data or demonstrate generalizability of qualitative findings. Barnes and colleagues used exploratory sequential mixed methods design to understand female surgeons’ gender bias experiences.²³ They first conducted focus groups with female surgeons to understand their experiences of sexism and microaggressions. Through analysis of the qualitative data, they generated survey questions that enabled them to quantitatively examine the microaggressions female surgeons experience.

In an explanatory sequential design, the quantitative phase precedes the qualitative. Explanatory design enables researchers to better understand quantitative findings through qualitative inquiry. This explanatory integration distinguishes a mixed methods study from separate qualitative and quantitative phases that address the same topic but are conducted independently. To examine the effect of coaching in graduate medical education, Parris and colleagues followed quantitative data collection in a randomized controlled trial with focus groups within each condition to understand the mechanisms of the effect of coaching.²⁸

In sequential designs, analysis occurs after each phase of the study. Integration entails describing how the qualitative findings informed creation of the quantitative measure (exploratory sequential) or how associations found in the quantitative analysis were more comprehensively understood using subsequent qualitative inquiry (explanatory sequential). These components also can be jointly displayed in tables²⁹ linking the qualitative and quantitative phases of the study. In these tables, a column should explain the data integration, or how data from the first phase of the study was made relevant to the second phase of the study.

In embedded designs, the qualitative or quantitative component plays a supplemental role within a larger study using the alternate approach. Frequently, embedded design includes a focused qualitative subcomponent within a larger study with quantitative outcomes. The timing of quantitative and qualitative data collection can vary; each phase can inform the other, or they can both try to answer the same research question (as commonly seen in convergent designs). For example, in Leal and colleagues’ pilot study examining the feasibility and efficacy of a yoga intervention, the researchers examined one domain of efficacy using simultaneously collected qualitative and quantitative data.³⁰ The remainder of the study was primarily quantitative.

For researchers considering mixed methods, we recommend the resources listed in Box 1.

To be successful, mixed methods researchers should prepare for three common challenges: working with a diverse team, taking their time, and managing an abundance of data.

Mixed methods requires a group of researchers who contribute a diversity of scientific perspectives and methodological and content expertise. At the conceptual level, challenges can arise with the language of the method alone. Often researchers have training in either quantitative or qualitative methods rather than primarily receiving training in mixed methods. Team members may feel limited by their lack of confidence or ability to perform in two methods. Differences in training also lead to differences in how scholars with different methodological backgrounds use and understand terminology. For example, the term “qualitative” can mean something different to quantitative versus qualitative researchers.

Beyond language, team members are likely to come from different research paradigms. Diverging paradigms can create conflict in teams. At the formation of a mixed methods team, researchers should have a conversation about how they will approach the research, creating a clear expectation for which approach is foregrounded. In medical education, the possibility exists that researchers have not considered their ontology or epistemology. Writing and sharing positionality statements can help team members understand one another’s perspective and inform team reflexivity.³¹ Team members should stay open to constructive conversation.

Time is a constant challenge in mixed methods research. The effort to mix methods often requires more than double the time to conduct either quantitative or qualitative methods alone. Mixed methods is not just the collection of two types of data that are summarized separately without showing how they inform each other. In addition to increased time for data collection (which can be longer for sequential designs that require completing analysis for one method before beginning the next), mixed methods research requires time for connecting and integrating data. Integrated analysis is cognitively challenging and requires time for deep thinking. Although computer programs can help organize data, discovering patterns and meaning through integration takes time.

Mixed methods researchers also frequently have an overabundance of data and findings. In a mixed methods study, more data does not equate to just adding more rows or columns in a spreadsheet. Rather, adding another data collection technique increases the type of data available to the researcher, such as observation notes or video recordings. In the same way, researchers have multiple options for integrating data sets. Before data collection begins, researchers should prepare comprehensive data management, data analysis, and dissemination plans. Data management and analysis plans, including data management procedures,³² help researchers focus and prioritize planned analysis; but in mixed methods, researchers are sensitive to emerging findings and how they can influence analysis plans.

In disseminating mixed methods studies, researchers must find the balance between depth and breadth in telling the research story. With the abundance of two types of data, researchers first face the decision of how much of a study fits in a single paper: How do researchers maximize impact and demonstrate rigor while ensuring reader understanding? Current norms do not provide a guidepost for this decision. Word limits³³ and editorial preferences structurally guide this dissemination decision, pushing researchers to prioritize a journal’s style and instructions³⁴ above the methodological considerations. Journal requirements do not regularly fit mixed methods reporting. As an example, the Journal of Mixed Methods Research has a substantially higher word limit—10,000 words for empirical mixed methods research studies³⁵—than most medical and medical education journals.

Additionally, the pressure in medical education to maximize publication³⁶ tempts researchers to divide studies into multiple papers. However, slicing apart mixed methods research dilutes the value and meaning of the work. If researchers choose to disseminate findings in multiple papers, they must decide in what order to write and submit multiple manuscripts for publication. Timing the publication of multiple papers that rely on (and cite) one another is complex.

Just as mixed methods teams are often diverse, authors should not make assumptions about readers’ training and experience. Writing clear methods sections helps readers understand the value and limitations of this work. The complex and varied options of mixing methods requires authors to clearly explain design decisions that are rarely explained in other methods such as experimental or survey methods. However, journals’ word limits often prevent detailed descriptions of mixed methods, leaving readers to make inferences about the approach (eg, making assumptions about authors’ research paradigms or methods rationales, as we did for some studies referenced in this paper). Ideally, mixed methods manuscripts should explicitly describe the theoretical frameworks, value of conducting a mixed methods study to answer the research question, researchers’ worldview and social positioning, and mixing and integration plan.

First, authors should describe the theoretical frameworks that contextualize the relevance of the research question and provide justification for the approach. For example, Lukachko and colleagues relied on Krieger’s ecosocial theory of racism³⁷—which holds that racism benefits the dominant racial group while simultaneously harming the oppressed racial group—to posit that state-level racism would be associated with higher likelihood of myocardial infarction among Black Americans.³⁸ Their use of this theory provided justification for their incorporation of stratified analyses to examine how racism differentially impacts White and Black people. Mixed methods research also requires authors’ engagement of theoretical frameworks to ground both the qualitative and quantitative components and provide an orientation as to how each of these components should be used to answer the research question.

Second, authors should clearly explain the value of conducting a mixed methods study. In Hirose and Cresswell’s criteria for quality mixed methods research,³⁹ two of their criteria relate to this aspect: (a) providing a rationale for using mixed methods methodology and (b) describing the quantitative, qualitative, and mixed methods aims of the study. The second component necessitates understanding why mixed methods are specifically needed to examine the research question. Understanding this need drives the research approach. A good example of describing the relevance of the mixed methods approach is the work by Rao and colleagues examining barriers to medication adherence.⁴⁰ They describe the importance of the qualitative and quantitative components, integrating their findings with each other. The brief Good Reporting of a Mixed Methods Study checklist⁴¹ is a guide for researchers preparing research reports of mixed methods studies that focuses on describing the rationale and integration of their methods.

Third, authors should elucidate their worldview, specifically articulating their ontology and epistemology. The American Psychological Association Publications and Communications Task Force Report⁴² and the Office of Behavioral and Social Sciences Research²⁶ document the importance of researchers describing their philosophy. Authors also should describe their social positioning (eg, culture, ethnicity, gender, race) as it relates to the research question. Positionality and reflexivity statements are instrumental in this effort. Although this criterion is more commonly associated with qualitative research, it is also essential to well-crafted mixed methods work that contains qualitative components. Selling and colleagues provided a useful example of these statements, explaining how their identities influenced their analysis.⁴³

Finally, mixed methods research requires a clear description of data mixing and integrated analysis. Authors first describe the mixing of quantitative and qualitative data collection and how they influenced each other. Then authors describe how they integrated analysis. In the design phase, researchers generally create a design flow chart to guide the timing and integration of mixed methods data collection and analysis. Researchers should consider including this graphic illustration in mixed methods reports to more clearly communicate the process of timing and integration. Chandanabhumma and colleagues included a clear procedural diagram for their study on health care delivery.⁴⁴ Fetters provided a variety of design flow chart templates for researchers to consider.³²

Medical education—and medicine in general—has always leveraged both qualitative and quantitative insights to advance our collective knowledge around how to care for patients’ health. We recognize the value in researchers replicating the work of O’Cathain and colleagues⁴¹ to develop more comprehensive mixed methods reporting guidelines that reflect the advancements in the field since they conducted their review more than 15 years ago.

As mixed methods research continues to expand and advance, its conduct will change with time and innovation. For example, even though the pragmatist underpinnings of mixed methods research assume that any method could be used (so long as it is fitting for the research question), many mixed methods studies in medical education rely on survey and interview data. In the future, we encourage researchers to experiment with creative combinations of methods, such as EHR data, participant observation, and bibliometric analysis.

Additionally, we hope to see continued expansion around who is included on mixed methods teams. Medical education research teams already draw from a variety of disciplines beyond medicine, such as psychology, sociology, and communication. We encourage teams to reach even further. Including service design⁴⁵ and user experience⁴⁶ researchers in teams will continue to push the types of mixed methods research we can accomplish.

The authors acknowledge John Saultz and Nicholas LeFevre for their review and feedback on an earlier version of this manuscript.