Inappropriate Survey Design Misstates Findings on Use of Preference Signaling

The article “Use of Signaling in Family Medicine Residency Interviewing” discusses the new preference signaling process during the residency application cycle.¹ The survey permitted respondents to choose only one choice among multiple options, but in several questions more than one answer could have applied. Inaccurate assumptions could be drawn from the results. Incorrectly implying significance, or lack thereof, of preference signaling may negatively impact applicants' decisions and how they approach the application cycle.

This article pulled conclusions from a survey the authors created; however, the methodology of the survey was flawed. The survey allowed responders to choose only one answer on questions that should have permitted multiple options. For example, the question, “If an applicant signals your program, you would . . . .” has overlapping answer options, such as program directors considering the preference signal in a holistic review process but also automatically offering the candidate an interview if they meet other criteria. Best practice for “closed-ended questions should include all reasonable responses (ie, the list of options is exhaustive) and the response categories should not overlap (ie, response options should be mutually exclusive).”² Answer options should also be mutually exclusive and should ensure only one answer could apply rather than multiple.³ Further supporting the faulty nature of the questions, 8% of responders left nonbinary questions blank.

The residency match process is a pivotal moment for applicants. Overstating or understating the importance of preference signaling by interpreting a flawed survey can hinder this process. By having more open-ended responses, allowing the ability to choose multiple answers, and gaining applicant perspective on the process, this survey could have obtained helpful information. Program directors need to consider how applicants will view this data when making important decisions regarding preference signaling.