Prioritizing Research Metrics in Family Medicine: A CERA Study of Department Chairs

Building research capacity within family medicine has been recognized as a core component of the specialty’s growth and effectiveness. For more than 40 years, leaders in family medicine have highlighted the importance of expanding research capabilities to address the unique needs of primary care.^1-3 In recent years, this need has become more urgent, prompting initiatives such as the Family Medicine Research Summit held October 2023 in San Francisco.^4-6

Family medicine departments comprise a foundational piece of the specialty’s infrastructure. They support medical students, residency programs, and fellowships that provide foundational training for the next generation of family physicians, and they conduct most of the funded research within the specialty.⁷ Building research capacity within these departments is therefore necessary in order to support and grow the overall capacity of the specialty.⁸ Researchers have identified several key strategies to increase departmental research capacity, such as allocating protected faculty time, strengthening statistical expertise and grant-writing expertise, and fostering interdisciplinary collaboration.^9,10 Moreover, adding a PhD-level researcher, when one is not already present, has been shown to effectively bolster research capacity.¹¹

Family medicine departments, however, face a unique set of challenges because they possess multiple, often competing missions within health care systems.¹¹ Family medicine department chairs exert substantial influence over their department’s resource allocation and strategic priorities, balancing the often-competing tasks of clinical productivity, education and training, and research development.¹² Various factors, such as regulatory requirements, personal research experience, and institutional priorities, influence chairs’ perspectives on research and capacity building, shaping how they use resources.^12-14

Despite wide consensus that building research capacity is an important task for the specialty of family medicine, no clear standard exists for how to define or measure that growth. In a prior study, we conducted a scoping review that identified 31 quantitative metrics useful for assessing research capacity. These were organizational in scope and rated as highly relevant to a population of research leaders and department chairs. These metrics were grouped into seven categories: bibliometrics, impact, ongoing research, collaboration activities, funding, personnel, and education/academics. This framework, termed the Productivity and Capacity Evaluation in Research (PACER) Tool, comprises a list of 31 quantitative research metrics viewed as impactful and relevant by research leaders.⁵

In the initial Delphi process used to refine the PACER Tool, the panel of experts broadly agreed that each of the metrics was relevant in some way. However, a common critique was that obtaining all the data within the tool was impractical. Furthermore, no strong consensus existed regarding which categories of research metrics was most important to family medicine departments. To address this gap and clarify the priorities of quantitative research metrics in the field, we surveyed family medicine department chairs to assess which categories of PACER Tool metrics they considered most important. The primary outcome of this survey was to understand the types of research metrics family medicine department chairs considered most valuable in evaluating research productivity and capacity. A secondary aim was to compare their reported research capacity with answers given by a population of family medicine department chairs in 2021.

This survey was part of an omnibus study conducted by the Council of Academic Family Medicine Educational Research Alliance (CERA). The study utilized a cross-sectional survey design aimed at gathering information from family medicine department chairs in the United States and Canada. Methods for the CERA survey have been previously described.¹⁵ All survey questions were reviewed and approved by the CERA steering committee to ensure alignment with the subproject’s aims, readability, and evidence of reliability and validity. Pretesting was conducted among family medicine educators who were not in the target population to assess flow, timing, and readability. Approval for the study was obtained from the American Academy of Family Physicians Institutional Review Board in August 2024. The survey was conducted online using the online program SurveyMonkey (SurveyMonkey Inc) from August 13 to September 20, 2024. Weekly reminder emails were sent to minimize nonresponse bias, including five follow-up emails and a final reminder sent on the survey’s closing date. The target population consisted of all family medicine department chairs identified by the Association of Departments of Family Medicine (ADFM).

The first question in our survey was based on the ADFM Departmental Assessment of Research Tool (DART),¹¹ designed to help department chairs evaluate their department’s research capacity. The question prompted respondents to rate their department’s research capacity on a scale from 1 (extensive/replication research) to 5 (no or almost no research), with lower scores reflecting higher research productivity. To be able to compare responses with those given by department chairs in the 2021 CERA survey, the question answers used were verbatim from the prior survey.

We developed nine questions based on the categories of quantitative research metrics identified in the PACER Tool (bibliometrics, impact, ongoing research, collaboration activities, funding, personnel, and education/academics).⁵ The first seven questions asked respondents to rate the importance of each category on a 4-point scale (extremely important, very important, somewhat important, not important), and the final two questions asked them to select the most and least important categories (Supplemental Figure 1). To minimize bias and misunderstanding, the draft questions underwent pilot testing with family medicine clinicians, researchers, and department chairs. Feedback from pilot testing prompted minor changes to improve phrasing, and no concerns regarding question comprehension were identified.

We calculated basic descriptive statistics, specifically proportions for categorical data and averages (means) for numerical data, to summarize the survey results. Nonresponders for each question were not included in the analysis. Prior to comparing the answers to question 1 between 2021 and the more recent survey in 2024, we assessed normality in each sample to ensure the appropriateness of parametric testing, with ordinal data treated as interval data. Given differing sample sizes and the assumption of unequal variances, we performed Welch’s t test. Statistics were performed in Excel.

A total of 218 surveys were delivered to department chairs, with 203 (93.1%) sent to those in the United States and 15 (6.9%) to those in Canada. The survey invitation received 118 responses (54.1%), but 7 (3.2%) were abandoned and removed from the results because only the first demographic question was answered. This left 111 completed surveys (50.9%). The 2021 survey was administered using a similar process and resulted in 97 respondents with the primary endpoint data out of 192 invitations (50.5%).¹¹

See Table 1 for the demographic details of respondents. As noted earlier, the study population demographics closely matched the demographic information contained in the membership database data of ADFM.

With regard to our primary aim of addressing which categories of PACER Tool metrics department chairs considered most important, funding emerged consistently as the most impactful category, being rated as most important by the greatest number of respondents (42/108, 38.9%; Figure 1). Funding also was the category that was rated extremely important by the greatest number of department chairs (39/108, 36.1%), followed by personnel (27/108, 25%; Figure 2).

Subgroup analyses separated departments into categories based on quartiles of staff full-time equivalents (FTEs) and DART research capacity level. Interquartile ranges were 18 or fewer FTE for the first quartile (n = 27), 19–31 for the second quartile (n = 26), 23 to 74 for the third quartile (n = 24), and 75 or more for the fourth quartile (n = 27). Respondents that did not provide data on department size (n = 5) or DART level (n = 2) were excluded from the respective subgroup analyses.

Funding was rated as most important by more than half of respondents in the fourth (n = 14/27, 51.9%) and third (n = 13/24, 54.2%) department size quartiles (Table 2). Funding was also rated as most important by more than half of respondents who placed themselves in the highest levels of research capacity (extensive/replication research and significant/self-sustaining research, n = 24/38, 63.2%). Among chairs representing smaller departments (quartiles 1 and 2), funding was the most common category rated as most important, though less than half rated it as most important (n = 15/53, 28.3%). This finding also was true for department chairs reporting lower levels of research capacity (n = 18/70, 25.7%; Table 2).

While funding was consistently the most important category, no single category was consistently rated as the second most important. Additionally, no clear trend existed in the least important category, whether distinguished by department size or by reported research capacity.

Among the various categories of quantitative research metrics, bibliometrics was most split among opinions. It received the highest number of responses for the least important category in the survey, while also receiving the second highest number of responses for the most important. When we further examined this data by reported research capacity, departments with a higher research capacity reported a low importance of bibliometrics, while departments with lower research capacity reported a relatively higher importance of bibliometrics.

The mean value of reported research capacity level on the DART changed from 2.98 in 2021 to 2.90 in 2024, with lower numbers representing more research capacity (Figure 3). This difference was not statistically significant (P=0.61).

This study aimed to discern which category of quantitative research metrics are most valued by family medicine department chairs. Across diverse institutions, family medicine department chairs consistently identified funding as the most important metric category. Larger departments and those with greater reported research capacity were more likely to prioritize funding. This emphasis on funding likely reflects its critical role in sustaining research infrastructure, staffing, and departmental stability, especially in larger departments that are reliant on consistent grant support. Indirect funds from grants offer essential discretionary resources for future research investment. Smaller departments and those with less reported research capacity may emphasize other metrics, such as bibliometrics, as a strategy to chart foundational steps toward securing external funding. These differing priorities may account for the relatively lower importance assigned to funding.

Results were mixed regarding the relative importance of the remaining categories of quantitative research metrics. While each of the categories contained in the PACER Tool may be important in the understanding of a department’s research capacity,⁵ the current findings suggest that different department chairs view the relative importance of the various categories quite differently from one another. No single category was consistently rated as the second most important, and no category was consistently rated as the least important. After funding, the second most common category to be labeled as most important was bibliometrics. However, it was also the category that was rated least important by the greatest number of respondents. Impact was the category that was rated as not important or only somewhat important by the greatest number of respondents. However, this finding was not consistent with the categories rated by respondents as least important.

Notably, each category was viewed as very important or extremely important by more than 50% of all respondents, and no category was viewed as not important by more than 10%. These results suggest that those wishing to communicate the value of their research enterprise to family medicine department chairs should include funding as a category of metrics to include and select other quantitative metrics based on the preferences of the chair and needs of the department. Prior research suggested that for funding information to be most impactful, it should be contextualized by other factors such as bibliometrics and impact.^5,16 Funding is not necessarily an end in itself, but a means to other ends, many of which also can be measured. However, no current benchmarks exist that normalize other data, such as the number of publications a department should expect per dollar spent on research or whether increased spending on research leads to increased impact of published articles. If the research metrics contained in the PACER Tool were to become widely adopted, further research could explore these areas.

The results of this study are limited because they reflect only the views of respondents. Nonresponse bias may alter results if, for example, departments with less research priority were less likely to respond. Additionally, CERA surveys allow a limited number and type of questions, precluding use of survey techniques, such as ranking order, that may be more appropriate to the question. Survey length requirements also limited the number of possible responses, reducing the ability to make fine distinctions between categories.

Further research would be required to establish whether these results apply to other populations that affect family medicine research, such as deans, chairs of research divisions, clinicians, and primary researchers. Also unknown is how well these results may reflect the attitudes of leaders in other specialties. Notably, this survey did not explore individual metrics within these categories; for example, publications and presentations are each separate metrics within the category bibliometrics. Also, while the survey included definitions of the seven investigated categories (bibliometrics, impact, ongoing research, collaboration activities, funding, personnel, education/academics), varied understanding of what types of metrics were included in each of the categories may have influenced the results. Efforts to minimize bias and misunderstanding included rigorous pretesting on individuals within and outside of the target population, though further research may focus on in-depth interviews of the target population.

In a landscape of limited grant availability, rising competition for research support, and an unpredictable regulatory environment, these findings underscore the central role of securing research funding for academic family medicine departments. To support research growth, department chairs may consider adopting strategies such as strengthening grant-writing support, fostering collaborations, and advocating for institutional investment in research infrastructure. Targeted efforts to improve funding mechanisms may enhance research productivity and reinforce the role of family medicine in advancing primary care and population health.

Funding emerged as the category of research metrics viewed as most important by the greatest number of family medicine department chairs. Other categories of metrics also were viewed as important, though no clear pattern emerged regarding their relative importance in this population. Broader adoption of tools like PACER may enable more nuanced evaluation of research productivity and inform strategic resource allocation.