A Descriptive Bibliometric Study of CERA Publication Dissemination, Authorship, and Citation Rates

Several barriers impede the production of high-quality research in family medicine; these barriers include shortages of funding, infrastructure, and mentorship. ^{1, 2} In 2011, the Council of Academic Family Medicine Educational Research Alliance (CERA) was created to set a standard for family medicine medical education research that is rigorous and generalizable. CERA provides mentoring and education to junior researchers, facilitates collaboration among medical education researchers, and guides the specialty by providing leadership and vision for medical education research. ³

CERA was created with three components.³ First, it provides the infrastructure to collate survey-based research. This infrastructure is all-inclusive and includes question writing, survey dissemination, and data organization.⁴ Second, CERA provides faculty experienced in survey-based research to mentor researchers. These faculty members are available to the research team to optimize survey questions, analyze survey responses, and write conference abstracts and peer-reviewed publications. ⁵ Third, CERA maintains a data repository of all survey results available to Society of Teachers of Family Medicine (STFM) members. This database serves as an opportunity to perform secondary research analyzing associations among questions from different surveys. ⁶

From its inception through 2023, CERA has deployed 62 omnibus surveys, mentored numerous early-career researchers, and created a rich repository for secondary research. CERA has proven to be an important infrastructure for the family medicine research community.⁷ Although CERA lists many of its related publications and presentations, a rigorous analysis of the scope and impact of CERA in the published literature had not yet been done. ⁸ We sought to perform a bibliographic analysis that describes the scope and impact that CERA-related publications have had to date.

We performed a descriptive bibliometric study to describe the scope of CERA-related publications in terms of the diversity of authors, institutions, and journals as well as the impact of these publications. ⁹ We chose this methodology with the goal of obtaining meaningful insights that could help direct the future of CERA-related publications. Because of the diversity of the topics explored by CERA survey research, we did not feel that an explorative bibliometric study would effectively answer our question.

We obtained articles related to CERA survey infrastructure by searching Medline (PubMed) and SCOPUS using the terms “CERA” and “survey” for the period 2011 to 2023. We examined reference lists of all included articles to identify additional publications. We also used the CERA website, which lists most publications related to CERA survey infrastructure. Included articles were peer-reviewed, published research or editorials in English that described or used survey infrastructure provided by CERA. We reviewed the abstracts to ensure that each study we included discussed or described CERA surveys and data.

We evaluated articles based on authors, affiliations, date of publication, citations, publication type, and CERA survey type discussed. Additionally, we evaluated the Field-Weighted Citation Impact (FWCI) from SCOPUS. Field-Weighted Citation Impact indicates how the number of citations received by an article compares to the average number of citations received by similar publications. We used descriptive statistics to describe percentages of total CERA-related publications, total CERA-related citations, and citations per publication. Data were organized using Microsoft Excel.

We retrieved a total of 231 articles from our original search, and we retrieved 166 articles from the CERA website. A total of 57 articles were excluded because they did not relate to CERA survey infrastructure. The acronym CERA in those excluded articles referred to Centre for Eye Research Australia or Continuous Erythropoiesis Receptor Activator. A total of 174 articles were included in the analysis. From the 174 included articles, 127 articles were found from both sources of data, 8 articles were found through the literature search only, and 39 articles were found on the CERA website only. The first article was published in 2011 (Figure 1). The number of annual publications related to CERA increased and then plateaued to a mean of 17.7 publications yearly. Most studies (96.0%, 167/174 publications) were original research with 98.2% (164/167 publications) consisting of primary research studies and 1.8% (3/167 publications) consisting of secondary analyses. The remaining articles were either editorials (2.9%, 5/174 publications) or described methods (1.1%, 2/174 publications). CERA-related publications were found in 24 unique journals. The most common journals publishing CERA-related articles were Family Medicine (69.5%, 121/174 publications), PRiMER (8.6%, 15/174 publications), Journal of the American Board of Family Medicine (5.7%, 10/174 publications), Annals of Family Medicine (2.3%, 4/174 publications), Journal of Graduate Medical Education (1.7%, 3/174 publications), and Academic Medicine (1.7%, 3/174 publications). Eighteen other journals had one CERA-related publication. The most common CERA survey articles were about the program director survey (50.6%, 86/174 publications), followed by clerkship director (23.5%, 40/174 publications), general membership (15.9%, 27/174 publications), and department chair (6.5%, 11/174 publications). The resident and medical student surveys had 2.9% (5/174 publications) and 0.6% (1/174 publications) of publications, respectively.

In addition to total citations, we evaluated the FWCI, which is a measure of dissemination normalized to topic and field of study. The average FWCI for all CERA-related publications was 1.08±0.1, indicating no significant difference in dissemination beyond that of other articles on similar topics. FWCI was higher for CERA editorial (2.08) and methods (3.51) publications than CERA original research (1.04). The average FWCI was highest for articles from general membership surveys, which had an FWCI of 1.72, followed by publications related to program director surveys (0.99), resident survey (0.98), clerkship director surveys (0.86), and medical student surveys (0.69).

Expectedly, publications prior to 2017 had significantly more citations per publication (average of 6.2 citations per publication) than publications after 2017 (average of 2.4 citations per publication; Table 1). Editorial and methods publications had more citations per publication (7.0–8.0 citations per publication) than original research publications (3.1–3.3 citations per publication). Although the journal Family Medicine published the most CERA-related publications, it also had the lowest citations per publication. When evaluating the CERA survey type with the most citations per publication, general membership surveys had 6.3 citations per publication, department chair surveys had 3.1 citations per publication, and program director surveys had 3.0 citations per publication.

The top 10 most cited CERA-related publications are listed in Table 2. This included 8 original research articles, 1 editorial article, and 1 methods article. Of the original research publications on this list, 4 publications reported on general membership CERA surveys, 2 publications reported on program director CERA surveys, and 1 publication each reported on clerkship director and department chair CERA surveys.

We identified 515 unique authors on CERA-related publications. The top 10 most common authors of CERA-related publications were Kelly Everard with 20 CERA-related publications, Arch Mainous III with 19 CERA-related publications, and Dean Seehusen with 18 CERA-related publications (Table 3 ). Arch Mainous III had the most citations from CERA-related publications, with 182 citations and 9.6 citations per publication.

We found 153 unique affiliations associated with the authors of CERA-related publications. The top 10 most common institutions producing CERA-related publications are listed in Table 4. University of South Carolina had the highest number of affiliated authors publications and citations at 56 authors, 30 publications, and 177 citations. University of Florida had the most citations per publication at 7.1 citations per publication.

CERA was developed with three main components: (a) infrastructure for survey-based research, (b) mentoring of early-career researchers, and (c) creation of a data warehouse for secondary research. ³, ⁶ We sought to perform a programmatic assessment of these components to describe the scope and impact of CERA-related publications in the peer-reviewed literature.

CERA has been instrumental in accelerating the progress of family medicine education research by providing a structured platform that facilitates high-quality studies and disseminates original research findings. Moreover, CERA’s mentoring and resources for inexperienced researchers have been invaluable, offering guidance and support that helps budding researchers develop their skills and contribute to the field. Additionally, the insights provided by CERA’s original research have significantly informed and enhanced the practices of family medicine educators, ensuring that educational strategies are grounded in evidence-based research.

The findings suggest that editorial and methodology publications related to CERA are the most highly cited. Despite being infrequently cited, original research, especially primary research studies, remains the most common type of CERA-related publications. We also found that general membership CERA surveys were the most highly cited, followed by program directors, department chairs, and clerkship directors. Resident and medical student CERA surveys did not have enough publications to make assumptions regarding their placement on this list. General membership CERA surveys more commonly researched clinical topics than medical education topics, which could explain their higher citation rates.

CERA-related publications were almost exclusively found in family medicine journals, namely Family Medicine and PRiMER (78.1% of publications). Family Medicine has a 5-year Journal Citation Report (JCR) impact factor of 1.80; PRiMER has yet to acquire a JCR impact factor. The journals with the highest impact factor publishing CERA-related publications were Annals of Family Medicine (5-year JCR impact factor=5.71) and Academic Medicine (5-year JCR impact factor=5.35), both of which had few publications (<5% of publications). We found that CERA-related publications, on average, were disseminated similar to other publications of the same/similar topic. Editorial and methods publications were found to have higher FCWI than original research publications, indicating that they were disseminated more than similar publications in their fields.

We found that CERA-related publications were written by 515 unique authors from 153 academic institutions. The authors most commonly writing CERA-related publications have served as CERA mentors for early career researchers in the creation, analysis, and dissemination of surveys. Despite the large network of unique authors, a coauthor social network analysis would be beneficial to determine the strength of the CERA mentorship program. The institutions most involved can be matched with many of the most common authors of CERA-related manuscripts.

CERA has created a repository of all CERA survey research questions that are publicly available on the STFM website to STFM members. ⁶ Our study showed that only 3 publications were secondary analysis research out of 167 research publications; the CERA database is an underused resource for scholarly activity. Over the past decade, many of the demographic questions used for the surveys have changed, which makes longitudinal data analysis difficult. ¹⁰ Increasing opportunities for longitudinal and secondary data analysis is the greatest opportunity for improvement within CERA’s three main pillars. ³ One solution would be to de-identify the codes used to track survey respondents and their program affiliation across years; this de-identification would allow for analysis of surveys between years. Another solution would be to organize raw survey data to be more easily interoperable than the current format of individual spreadsheets.

CERA provides an important and valuable infrastructure for performing family medicine research, which has led to a great diversity of authors and affiliations. CERA has made a significant impact in family medicine journals but has not had reach outside of the discipline. The most impactful CERA-related survey type is the general membership survey, which has focused more on clinical topics rather than medical education topics. CERA support should continue while encouraging efforts to expand its impact beyond family medicine journals.