Overview of Quantitative Research

Quantitative research is a structured, empirically grounded approach that goes beyond simply gathering numerical data; it employs theoretically informed methodologies to test hypotheses and investigate relationships, patterns, and trends. Quantitative research offers various approaches one can choose to answer the research questions and understand the data, yet each approach comes with distinct advantages and limitations. This overview provides a general introduction to quantitative research and various approaches to a quantitative study, as well as discusses common pitfalls and strategies to address them.

Quantitative methods are used when researchers aim to measure variables, test hypotheses, or evaluate relationships and outcomes—particularly when the goal is to determine how much, how often, or to what extent something occurs. Quantitative approaches are appropriate when researchers seek to generate measurable evidence about learning processes, performance, or program effectiveness. In practice, quantitative studies often address three types of questions: (a) descriptive, which summarize collected data in meaningful ways (eg, reporting average patterns from program evaluation data or quality improvement project); (b) causal, which estimate treatment or an intervention’s effects (eg, examining the impact of a new curriculum on learner outcomes); and (c) associative, which describe relationships between variables (eg, exploring how clinical competency is associated with trainees’ professional identity or values of professionalism).

Once the type of question is established, researchers must consider the intended scope of their findings—whether the goal is to improve a local program or to produce results that can inform broader educational contexts. Different choices of study design determine how results are interpreted. For example, a program may conduct a small evaluation study to inform curricular changes within its own setting. Such descriptive studies are valuable for identifying local trends and guiding program improvement, even though their findings are not intended to be generalized beyond that context. In one study, researchers evaluated family-oriented (FO) attitudes and observed skills of family medicine residents before and after a 20 week psychosocial medicine curriculum.¹ They found that stronger FO attitudes and modest increases in related behaviors were identified following the curriculum, which can be used to guide local curricular refinements.¹ Similarly, another study analyzed survey data from graduates of a single family medicine residency program to examine factors associated with focused practice.² Their findings—linking postgraduate year three training completion to adopting a focused practice approach and ranges of services provided—offered important insights to local institution as they refined the training policy.

In other cases, researchers pursued studies aimed at generalizability—that is, producing findings that can reasonably be applied to populations or settings beyond the original study context. Generalizable studies typically require sampling strategies and designs that reduce the chance that findings apply only to one program and instead reflect patterns likely to hold true in other settings—a key aspect of external validity.³ This type of research often includes multisite quantitative studies that test whether similar curricular interventions yield consistent outcomes across settings, or instrument-development studies designed to broadly evaluate learner outcomes. For example, Dyrbye and colleagues developed a well-being index using data from 2248 medical students across seven institutions.⁴ Careful attention to sampling and psychometric testing ensured that the index could validly measure well-being across diverse medical school contexts and future learner populations.

Within quantitative research, researchers may consider using different approaches of study designs to answer research questions.

Experimental studies are used when researchers want to determine whether a specific intervention (eg, a curriculum) directly causes a change in outcomes. The most rigorous form of experimental design is the randomized controlled trial (RCT), in which participants—or groups such as residency cohorts or clinic sites—are randomly assigned to different conditions to balance preintervention differences between groups. Studies have applied RCTs to evaluate intervention strategies targeting physician burnout across specialties.⁵ In such a design, one group may participate in a new curricular or wellness program (the intervention condition), while another group follows standard training (the comparison condition). A strong experimental study is characterized by internal validity, meaning that any observed changes in outcomes can be attributed to intervention completion (ie, the independent variable) rather than to other confounding factors.⁶ This approach requires establishing temporal precedence, with outcomes measured after the intervention to confirm causality. By carefully structuring when, how, and in what form interventions occur, researchers can minimize confounding variables and reduce alternative explanations.

The success of an experimental design also depends on maintaining control over the intervention’s core elements, including consistency of content, delivery, timing, and structure. Because of this high level of control, experimental studies often can yield reliable conclusions even with modest sample sizes (eg, 30 participants per condition). However, the extent to which findings can be applied to other training programs—known as external validity—depends on how representative the sample and study conditions are. While experimental designs are powerful for establishing cause-and-effect relationships, their highly controlled nature may limit generalizability to family medicine learning or practice environments.

Researchers often work in complex learning and clinical environments where full experimental control is not possible. When researchers can introduce an intervention, such as a new curriculum, but cannot fully control variables in the setting (eg, year-to-year variations in student composition and academic performance, or when clinical sites differ in patient population, teaching approach, or faculty support), a quasi-experimental design may be appropriate. These designs share the logic of experimentation but differ from true experiments in the degree of control researchers can exert over the learning environment, participant assignment, or contextual factors. Randomization may even be included in some quasi-experiments, such as when intact groups (eg, residency cohorts, clinic teams, or clerkship sites) are randomly assigned to receive different educational interventions. However, researchers still lack full control over participant-level exposure and contextual variation across settings, which keeps these designs distinct from true experiments. In other situations, randomization is not feasible, and groups are determined by practical or institutional constraints (eg, residents assigned to different clinical sites).

The strength of such a design depends on how well confounding factors are measured and addressed. Carefully determining key measures, such as residents’ initial clinical reasoning scores or prior exposure to similar curricula, is essential to control for group differences and to minimize alternative explanations for the results. For example, a family medicine residency might implement a new peer-coaching model in one cohort and compare outcomes (eg, clinical reasoning scores or well-being) with another cohort not receiving the intervention, while statistically adjusting for prior performance, training experience, and burnout levels to rule out their influence.

At times, researchers are not attempting to establish causation but to describe how variables are associated in naturally occurring data, without researcher manipulation of exposures or conditions. For example, one might examine whether participation in mentorship programs is related to self-efficacy,⁷ whether residents’ sense of belonging predicts burnout levels, or whether faculty feedback frequency is associated with residents’ clinical confidence. Because no variables are manipulated, correlational studies are nonexperimental, and their estimates describe associations rather than causal effects.^8-10 Correlational designs are relatively easy to implement, practical, and accessible, making them popular among researchers. These studies are particularly useful in applied educational settings, where randomization and having control are often impractical but understanding relationships among training, attitudes, and outcomes can inform program improvement. This design requires careful model specification and rigorous statistical control (eg, including and controlling for demographic variables) to account for potential confounding factors and measurement biases.

A key consideration is how data will be collected—whether through self-report, observation, standardized assessments, or existing data sources. This consideration can be viewed from three perspectives: whether the data are self-reported, whether data collection involves interaction with participants, and whether data are gathered at a single point in time or repeatedly over a period. Cross-sectional studies collect data from a sample at a single point in time, providing a snapshot of existing relationships or characteristics. In contrast, longitudinal studies use repeated measurements to track changes within the same participants over time, offering insights into developmental or temporal trends—for example, how in-training examination scores in residency change over time.

The major difference between primary data and secondary data analysis is whether new data are being collected. Primary data collection involves gathering original data through methods such as surveys, experiments, or structured observations. Secondary data analysis, by contrast, uses preexisting datasets that have already been collected for another purpose. This latter approach is increasingly common in medical education research, where researchers leverage curated datasets—such as the Council of Academic Family Medicine Educational Research Alliance survey database, Association of American Medical Colleges graduation questionnaire, or institutional learner assessment repositories—to examine trends, test new hypotheses, or explore predictors of outcomes without collecting new data. Secondary analyses can provide high-value insights at lower cost and with faster turnaround while still allowing for robust quantitative inquiry.

Observation refers to a data-collection method in which researchers systematically record behaviors, interactions, or environmental features as they occur in natural or structured settings. Observational data collection relies on naturally occurring data (no researcher-participant interaction) to explore trends and patterns; for example, researchers have used video-based observation in primary care to analyze clinician-patient communication patterns.¹¹ Structured observation protocols and reliability checks (eg, interrater reliability)¹² help ensure consistency in how behaviors are categorized and interpreted.

Surveys are among the most widely used quantitative data-collection tools, in which structured questions are designed to capture participants’ self-reported attitudes, perceptions, or experiences through standardized response options or rating scales.¹³ One study illustrated this approach by surveying medical students about their confidence in addressing social determinants of health and their preparedness for advocacy.¹⁴ Recommendations also outline best practices for survey design and implementation.¹³ A systematic process—such as conducting a literature review, using interviews or focus groups to capture the target population’s language, obtaining expert validation on item clarity and relevance, and performing cognitive interviews to ensure intended interpretation—can strengthen both measurement quality and help reduce common errors in survey design (eg, ambiguous wording or inconsistent response scales).

As data-collection methods, both observational and survey methods can be implemented in experimental, quasi-experimental, or correlational research designs. Meanwhile, some researchers may use the terms observational study and survey study to describe a stand-alone research design in which these tools are the primary source of data (eg, a longitudinal survey study design).¹⁵

Quantitative research may be theory-driven or data-driven. A confirmatory approach aims to validate existing theories, whereas the exploratory approach seeks to identify new patterns or relationships.

When the goal is to validate and refine theories from other fields (eg, examine the effects of a curriculum intervention), a confirmatory approach is practical. In such work, the links among theory, hypotheses, and result testing guide study design and interpretation.¹⁶ For example, theoretical frameworks such as self-determination theory,¹⁷ cognitive load theory,¹⁸ and theories of identity formation,^19,20 offer a conceptual foundation for formulating hypotheses that predict learner outcomes. Hypotheses derived from established theories are then tested to determine whether empirical evidence supports or fails to support them. For example, in a study where experiential learning theory informed the research question, researchers assessed whether different teaching modalities—such as theater in education, simulated patients, and role-play—were equally effective in enhancing communication skills among second-year medical students.²¹ Another study used theoretical modeling to evaluate whether a self-directed learning curriculum improves clinical competency.²² This top-down, confirmatory, and theoretical-driven approach helps refine curricula and determine whether a prespecified learning model holds or needs revision.

Exploratory research identifies patterns or relationships when little prior knowledge exists. For example, in the same experiential learning study,²¹ researchers also explored gender differences in communication skills following different teaching modalities without a guided theory. This exploration allowed them to evaluate whether various teaching modalities work similarly well for learners across different demographic backgrounds. Exploratory results may also generate new hypotheses. For instance, one study analyzed survey data to examine correlations between preferred learning modalities and burnout among physician assistant students;²³ results revealed new areas for targeted interventions. Such findings can extend existing theories or serve as foundations for new models to be tested in later confirmatory studies.

Often, researchers use a mixed approach that combines exploratory and confirmatory methods—drawing on established predictors while allowing data-driven insights to emerge. This blended strategy enables testing theory-driven questions while remaining open to context-specific patterns that refine subsequent hypotheses. For example, a study on medical students’ coping strategies began with coping theory to guide analyses, then identified strategies unique to the medical school environment.²⁴

Reflecting on the limitations of quantitative research is important. While this research offers valuable insights, it faces challenges such as statistical misinterpretations, overgeneralization, and issues in measurement quality and social desirability bias.

A common issue is overgeneralization, where results from a specific sample (eg, students in one institution) are applied too broadly. Small-scale studies using primary data are particularly prone to this problem, but it can be mitigated by providing detailed demographic descriptions of the sample and contextual information (both historical and geographic) about the study site. This information enables readers to assess the transferability of findings to their own settings. Authors also should be careful not to overstate the broader significance of a single study’s results; conclusions should reflect the scope and scale of the evidence presented.

Overuse of causal language is another frequent issue when studies with observational or cross-sectional designs report findings in causal terms.^25,26 For example, a study might find that students who engage in active learning perform better on exams; but without an experimental design, one cannot conclude that active learning caused the improved performance. This error often arises from using terms like “impact” or “influence” to describe associations between variables. Even in longitudinal designs where surveys are administered repeatedly, variables measured earlier cannot be definitively interpreted as causes of later outcomes. Such studies are better described as longitudinal correlational, and terms like “longitudinal association” should be used. Researchers must align conclusions with study design and clearly distinguish correlation from causation.

A common misconception is the overreliance on P values to determine significance. A P value less than .05 does not prove that a hypothesis is true; rather, it represents the probability of obtaining a result as extreme as (or more extreme than) the observed one, assuming that the null hypothesis is true. In other words, a small P value indicates that such a result would be unlikely to occur by chance across repeated samples if no true effect exists. Likewise, a nonsignificant P value does not confirm the absence of a relationship; it may simply reflect low power, often due to small sample size. Instead of focusing solely on P values, researchers may report confidence intervals, model fit indicators, and effect sizes for a more nuanced interpretation of results.

Additionally, conducting multiple statistical tests without adjusting the study Type I error rate increases the likelihood of false positives. Some studies have reported dozens of significance tests, which inflated the overall Type I error (the probability of incorrectly rejecting at least one true null hypothesis). To mitigate this problem, ideally researchers would report all performed tests, include effect sizes,²⁷ and, most importantly, apply proper alpha corrections when multiple comparisons are conducted.^28-31

In research, unequal sample sizes often occur when collecting survey data in real-world settings. When comparing unequal groups (eg, 100 male participants vs 200 female participants), the statistical power of the analysis is primarily affected by the size of the smaller group, and thus the power to detect significant differences may be reduced.³² However, this difference in sample size does not imply that group comparisons are invalid.³³ Rather, the results remain interpretable and meaningful.

In survey-based quantitative studies, a recurring challenge is ensuring measurement quality. Two core aspects are reliability—the consistency of an instrument, and construct validity—the extent to which it measures what it is intended to measure. Problems arise when instruments conflate related but distinct domains. For instance, a survey intended to assess communication skill development may capture learners’ confidence or intentions rather than behavioral change, and a clinical competency scale may measure perceived proficiency rather than performance. Such mismatches highlight the need to critically evaluate whether an instrument aligns with the intended construct, rather than assuming that a previously validated tool always measures what it claims.

Even with existing measures, reliability and validity are not automatically transferable across contexts—a limitation referred to as measurement transferability. Because evidence of reliability and validity is tied to the context in which it was established, an instrument that performs well in one setting may not function the same way in another. For example, a professionalism scale validated with practicing physicians may not be equally reliable or valid for postgraduate trainees, and is even less appropriate for first-year medical students, whose developmental stage and interpretations of professionalism differ substantially. Thus, researchers should not rely solely on previously reported reliability and validity but should consider contextual differences and, when possible, evaluate instruments in their own setting.

Response biases are common in self-report measures. Social desirability bias³⁴ occurs when participants provide answers they believe will be viewed favorably by others. For instance, students may overstate patient-centered attitudes to meet perceived expectations of faculty. Acquiescence bias, by contrast, reflects a general tendency to agree with statements regardless of content.³⁵ Although distinct, these biases can overlap, especially when positively worded items align with socially valued behaviors. For example, residents might overreport patient-centered attitudes or professional values, either to align with perceived norms (social desirability bias) or simply to agree with affirming statements (acquiescence bias). To address these issues, researchers sometimes include social desirability scales as covariates,^36,37 but their effectiveness varies.³⁴ Balancing positively and negatively worded items, ensuring response anonymity, and using neutral phrasing also can help reduce acquiescence.

Triangulating self-reports with complementary data sources, such as clinical performance evaluations, patient feedback, faculty assessments, or direct observations may minimize these issues.^38-40 While multisource data (eg, 360 degree evaluations) can enhance research validity,^41,42 collecting that data is not always appropriate or feasible.^34,43,44 This complexity becomes profound where evaluating clinical competence requires input from peers, faculty, and patients; perspectives that may not always align.⁴⁵ For example, a resident’s self-assessment of their communication skills may differ significantly from a patient’s or a faculty’s perception.⁴⁵

While performance-based measures can reduce certain reporting bias, they may not fully capture the nuanced experiences central to educational, psychological, and affective constructs. In areas such as burnout, well-being, emotion regulation, or mistreatment, learners’ interpretations and personal perspectives provide essential insight into mental health outcomes (eg, depression).⁴⁶ Researchers should carefully design measurements that align with the constructs and the study goals. Because affective and behavioral domains evolve over time, longitudinal follow-up with repeated measures can strengthen internal validity by capturing temporal change and reducing reliance on single time-point self-reports.

Quantitative research has advanced medical education, but its value and applicability rely on rigorous design, measurement, and analysis that align with research questions. Treating different study designs (eg, experimental, correlational) equivalently and using statistical controls to compensate for weak study design can lead researchers to claim more than their data really show. Pressure to demonstrate impact often pushes researchers to use cause-and-effect language even when data only show relationships. Overreliance on P values and weak or misaligned measures further compound these risks. We urge researchers to be explicit about design logic, transparent about bias, and cautious when translating statistical associations into conclusions.

The views expressed in this manuscript are solely those of the authors and do not necessarily reflect those of the Uniformed Services University of the Health Sciences, the Henry M. Jackson Foundation, or the United States Department of War.