Captive Samples Are Not the Answer to Survey Response Rates

I was eager to read Ericson et al’s “Optimizing Survey Response Rates in Graduate Medical Education Research Studies,” which described potential strategies to optimize response rates to survey research. ¹ Unfortunately, the authors’ described strategies are not generalizable because of their highly motivated sample populations. The residency programs involved in the Preparing the Personal Physician for Practice and Length of Training Pilot studies went through rigorous application processes to participate and fully understood the expectations for studying their experience. As such, the strategies presented lack utility due to their captive samples and likely social desirability biases.

Survey response rates in medical education research have steadily declined. ¹ Despite this decline, though, the three highest-impact journals for medical education (Academic Medicine, Medical Education, and Advances in Health Sciences Education) do not require a minimum survey response rate in studies submitted for publication. ² This lack of a requirement may stem from prior research suggesting that response rate does not inversely correlate with nonresponse bias and that justification of the target population sample better determines the relevance of the data. ³ In the article, the primary factor identified by the authors to increase response rate was the relationship between the participants and the investigative team; this relationship can reasonably be assumed to align with the motivation level of the respondent. ⁴ Because these programs were selected to participate in two large-scale longitudinal studies, social desirability bias likely contributed to the survey completion rate meeting the expectations from the evaluation teams. Given these factors, the external validity of this study is low. This concept is demonstrated well in the case of national Council of Academic Family Medicine Educational Research Alliance (CERA) surveys, which yield lower response rates with meaningful results for family medicine education. ⁵

This article showed promise in addressing survey response rates, a known gap in survey research and one of the primary methods of data collection in family medicine education research. Assessing a captive sample at high risk of social desirability bias, however, could not yield results generalizable to the wider family medicine community. To better support the relevance of survey research, peer-reviewed journals and research societies should adopt standardized guidelines for developing, analyzing, and reporting survey research data.