Become a PRiMER Reviewer
If you are interested in reviewing for Peer-reviewed Reports in Medical Education Research (PRiMER), please fill out our potential reviewer form. The information you provide will help the editors of PRiMER match reviewers with appropriate manuscripts.
The editorial team will review your information and get back in touch with you shortly.
Reviewers are encouraged to consider the following 10 comments about our review process. Some of these principles are specific to PRiMER, but most apply to any journal reviewing experience.
- Author Instructions and Quality Guidelines: As a teaching journal, PRiMER encourages new authors to submit manuscripts. PRiMER accepts early pilot studies, studies with small sample sizes, and projects that replicate or confirm existing research, as long as they meet the journal's quality standards. It is a priority for new reviewers to read all five tabs on the PRiMER “Author” page, at https://journals.stfm.org/primer/authors/ (Author Instructions, General Guidelines, Manuscript Preparation, Submit a Manuscript, and Quality Guidelines). The same guidelines are relevant to both authors and reviewers, and of these, the most important for a new reviewer is the "Quality Guidelines" tab (which also appears here in the PRiMER "Reviewers" page). These guidelines introduce the points we are asking reviewers to evaluate for each manuscript. If you are generally familiar with all of the author submission information, it will help you understand the mindset of the authors as they prepared and submitted their manuscript for review.
- Reviewer Instructions (and Quality Guidelines): Having read the quality guidelines and the other sections of the Author page, you’ll then be well prepared to read the "Reviewer" page, in particular, the "Style of Review” tab.
- Constructive Criticism: The information above is specific to PRiMER—although our Quality Guidelines are useful for any medical education scholarly writing. The PRiMER editorial staff strives for a tone of mentoring encouragement in our engagement with authors. Reviewers are not expected to overlook clear problems in a manuscript. However, constructive criticism is essential. It is not uncommon for new authors to be treated harshly and even subjected to personal attacks in the course of revision. This is not the PRiMER way. PRiMER reviewers have access to a "Confidential Comments for the Editors" feature in the review form, intended for sharing feedback to the editors about the paper that you may not want to share with the authors. Although this field is currently required, we encourage you to use it only when necessary, entering “not applicable” or “N/A” when not needed. If you have something particularly harsh to say, the confidential comments are where to get that out—the editors will take it under advisement.
- Length: It is not uncommon for a good, in-depth review to be nearly as long as the 1,000-word manuscript it pertains to. While there is no specific word minimum or limit for reviews, the review should be long enough to be informative for the editors to gain good advice, and the authors to gain instructive, constructive criticism, but short enough that this is not an onerous task for the reviewer.
- Audience for the Review: Bear in mind the first audience of your review is the associate editor and editor-in-chief for PRiMER. You are making a recommendation about the manuscript you have been invited to review to them, and they use your input, and the input of others, to make a decision about next steps. Secondly, the authors are also an audience. You are telling them in a constructive way what is good and what is flawed about the manuscript, and in a manner that allows the authors to remedy specific issues, if it is possible to do so.
- Structure of a Review: The best reviews are structured with a brief introductory paragraph, that simply states an overarching observation about the topic of the paper and its importance, and the overall quality of the study. A good review then breaks comments into distinct units of critique, each of which are individually amenable to evaluation by the editors, and individually correctable by the authors. For example:
This manuscript describes the evaluation of a new module implemented online for use in a family medicine clerkship at one institution. The authors focused upon the proximal effects of the online module on SHELF examination scores, as well as the distal effect upon [add distal effect], and reactions to the module via survey and focus group. The authors have done a reasonable job of presenting the work, and the evaluation appears to be fairly rigorous. However, I have several concerns that I believe would need to be addressed before publication:
a) The authors have done a fair job of connecting with the existing literature on $$$$$$, but they have left out several recent references that speak directly to the current activity. They should connect this paper with the work of @@@@@, either in the introduction or conclusion.
b) It is unclear whether there was any validation process that occurred with the survey, how it was administered, etc. There should be an additional paragraph describing.......[whatever you think is missing]
c) I am not certain the conclusion that ####### is really something the authors can draw from the analytic results they have presented. I would suggest that the conclusion be...
d) The authors have not considered the following limitations, and should describe these issues as additional limitations:...
e) Etc. Etc.
A review structured in this manner informs the editorial team of what to look at, and informs the authors what is wrong and needs correction, which is especially important if they are invited to revise and resubmit their manuscript for further consideration. The above example does not need to serve as a template—do what you think fits for your review. The example is a suggestion of what works for those on the receiving end of a review.
- Key Elements for Focus: A review should not focus on grammar or spelling. It is fine to note an occasional grammatical, typographical, or spelling error, but your primary role is to review the scientific, medical, and educational content. If a paper is poorly written, simply state that the writing needs improvement, and could use copyediting. Reviewers are simply expected to complete the reviewer form, entering comments into the appropriate text boxes. So, what then are the key elements to focus upon? Following are 10 key things to look for:
a) Does the introduction adequately describe a rationale for the study or project described in the manuscript?
b) Does the (sufficiently brief) literature review adequately build a connection to related papers, and offer evidence for why the study or procedure was conducted?
c) Does the introduction describe the research question(s) or hypothesis tested in the study?
d) Does the Methods section adequately describe all methods, including sampling, recruitment, data collection, analysis, and coding, as necessary?
e) If needed, are all approvals noted in the Methods section? (eg, Institutional Review Board (IRB) approval. We typically check this as an administrative step upon submission, but speak up if it looks like a necessary approval is missing)
f) Are the results clear and understandable?
g) Does the results section actually report all that the Methods section said was done? Does it report anything that was note mentioned in the Methods section?
h) Are the conclusions reasonably drawn from the results?
i) Is there a sufficient statement of limitations of the study? These help frame how a study should be interpreted by readers.
j) Are broader lessons described? Why should anyone aside from the authors care about the result? What can anyone else learn from the study?
- Invitations: Once you have become familiar with the guidelines above, you will likely soon receive an invitation to review a manuscript for PRiMER. It is never our intention to bombard reviewers with multiple reviews in succession, and it is never our intention for you to receive two reviews at once. However, accidents happen. Just let us know. The best thing you can do is to respond quickly to reviewer invitations, even if your response is to decline. You don't have to review manuscripts, but responding to the invitation quickly lets the editors know whether they need to seek another reviewer, and moves the process forward. The PRiMER editors generally expect to get reviews back within 2 weeks. Occasional delays are understandable, as long as they are communicated.
PRiMER reviewers are expected to declare their conflicts of interest and recuse themselves from the peer-review process if a conflict exists for a manuscript they have been invited to review.
- A Bit More About PRiMER: The invitation to review will likely come from an associate editor. Our full managing editorial team is listed here: https://journals.stfm.org/primer/editorial-team/. You can also read more about the mission and founding of PRiMER here: https://journals.stfm.org/primer/2017/morley/.
- Questions and Further Development: There will likely be a reviewer development session to be held at the next STFM Annual Spring Conference. There is no obligation to attend, but be on the lookout if you are going anyway. In the mean time, feel free to reach out to any member of our editorial team with any questions you may have.
Quality Guidelines for Authors and Reviewers
Manuscript Quality—PRiMER seeks to publish papers that rigorously evaluate educational interventions and learning outcomes or test for behavioral changes resulting from an educational intervention. PRiMER will consider small studies, pilot projects, single-institution studies, work that seeks to replicate or confirm findings that are already known, or research that explores a broadly known construct in a new context. Papers published in PRiMER must contribute knowledge that incrementally adds to what is known about a topic or phenomenon. We will consider quantitative, qualitative, and mixed-methods submissions that are sufficiently rigorous.
We generally will not publish reactions to new curricula or interventions (eg, surveys of learners about satisfaction with a course) unless the learning modality or instructional content is new, there is generalizable knowledge to be gained by instructors at other institutions, and key portions of the instructional materials are made available (typically by references to a stable format for publication of online content, such as the STFM Resource Library).
Manuscripts outside the scope of medical education may also be considered for PRiMER publication if the first author is a medical student or a family medicine resident, and the content is pertinent to family medicine as a broad discipline. Resident and student manuscripts will be held to the same quality standards outlined below.
For a general overview of research methods, consider this free online methods guide: The Research Methods Knowledge Base (https://www.socialresearchmethods.net/kb/)
The following elements should be included in all submissions when applicable:
- Data sources must be fully identified
- Human subject recruitment procedures must be described, if applicable, including:
- Institutional Review Board (IRB) interactions (see statement below)
- Consent processes
- Participation incentives
- Recruitment procedures, description of outreach, etc.
- Description of the sample characteristics, and as appropriate, a comparison of the sample to the population the sample should represent, or from which it is drawn
- Limitations: Manuscripts are expected to have a comprehensive description of all limitations, to allow complete assessment of quality by reviewers, and fully informed interpretation by readers. The statement of limitations is typically included as part of the Discussion section.
- A statement about an interaction with an IRB should be present in most studies, except as noted below. As a general rule, studies that utilized publicly available data (eg, public CDC data sets, reviews of published literature, observation of public activity, opinion or theory pieces, etc) do not require any IRB interaction. All other studies require IRB interaction. All manuscripts that require an interaction with an IRB should stipulate which type of interaction occurred:
- IRB determination that a project is “not research” (eg, quality improvement or assurance) or “not human subjects research” (eg, biological samples of deceased individuals).
Example: “This project was determined by #### IRB to constitute a quality improvement activity, and not human subjects research.”
- IRB determination that a project is research, but exempt from review1
Example: “As an anonymous survey, this project was determined to be exempt from review by ### IRB, citing exemption #2.”
- IRB review (expedited or full review). Example: “This project was reviewed and approved by ### IRB.”
Manuscripts are expected to consist of the following structural elements, in order:
- An Introduction and literature review should lead to the study question(s). The literature review should include and refer to appropriate literature. There should be a description of the research question(s) and hypotheses to be tested or problems to be solved by the project.
- A Methods section should be appropriate to answer the study questions. The methods should predict all reported results (eg, if the authors say they conducted a t-test, the results of the t-test should appear in the results).
- A Results section should be directly tied to all steps in the Methods section.
- A Discussion section should explain the meaning of the results and help readers place the research findings in appropriate context. The Discussion section should not take “flights of fancy.” More information on writing an effective discussion section is available at: http://www.rcjournal.com/contents/10.04/10.04.1238.pdf. The Discussion section should also contain a clear and complete description of study limitations.
- References should be listed in standard format, and follow AMA style (consistent with Family Medicine).
- Appendices (optional): Material that might go into an appendix should be handled by having the author submit the item (a curricular description, survey form, etc) to the STFM Resource Library, and then cite the resource (along with all other citations). Reference #7 in this article is a good example: https://journals.stfm.org/primer/2017/prunuske-0002/.
Use of the Kirkpatrick Model of Assessment
Although manuscripts do not need to refer directly to the Kirkpatrick Model (KM) of Assessment,2 it is helpful for reviewers, associate editors, and authors to have a basic understanding of the four levels at which outcomes might be assessed, according to KM. Briefly, the four KM levels are:
- Level 1: Reaction—The degree to which participants find the training favorable, engaging, and relevant to their jobs.
- This level might be thought of as the typical conference feedback form, eg, a “post-only” assessment of how the learner “liked” or experienced a training or educational event.
- For PRiMER, we will only rarely publish reactions, and then only if the learning modality or instructional content is new, there is generalizable knowledge to be gained by instructors at other institutions, and key portions of the instructional materials are made available (typically by references to a stable format for publication of online content, such as the STFM Resource Library). Level 1 (reaction) studies are more likely to be considered if done as a rigorous qualitative process.
- Level 2: Learning—The degree to which participants acquire the intended knowledge, skills, attitude, confidence and commitment based on their participation in the training.
- This is often measured by a “pre/post” design, where students complete a process (such as a test or survey), are exposed to an intervention, and then resubmit the same test or survey immediately after. This model is often affected by maturation, by quick reinforcement or “teaching to the test,” and other threats to validity.
- For PRiMER, learning MUST be assessed beyond simple pretest followed by nearly immediate posttest. Examples of acceptable forms of learning assessment include:
- Performance on a standardized process (eg, United States Medical Licensing Examination [USMLE], Family Medicine Computer-Assisted Simulations for Educating Students [fmCASES], a regularly administered institutional examination, a periodic and regular survey, etc), and comparison with a nonexposed or differently exposed cohort (eg, comparison of students exposed to an intervention, vs unexposed, on board scores or an Objective Structured Clinical Examination [OSCE])
- Posttest at an extended interval, on a test or instrument that measures a construct that was intended to be affected by the intervention (ie, assessment of long-term retention of knowledge).
- Level 3: Behavior—The degree to which participants apply what they learned during training when they are back on the job. For example, if residents and faculty were given comprehensive education and feedback about opioid prescribing in a family medicine office, a decreased number of opioid prescriptions would be a change in behavior.
- Level 4: Results—The degree to which targeted outcomes occur as a result of the training and the support and accountability package. For example, if residents and faculty were given comprehensive education and feedback about opioid prescribing in a family medicine office, a decrease in patient emergency room visits for opioid overdose would be considered a result.
Although researchers should use a rigorous approach to educational assessment, some educational interventions will not achieve their goals, or will have unintended consequences. Performance, behavior, and results should be measured, but they may not improve. The journal welcomes submissions with unexpected or negative results.
For quantitative studies, consider:
- Is the number of units used in the sample study (N) very small? Manuscripts that try to do too much with a very small sample may be problematic. We do accept small studies, but they should be handled appropriately for their sample size. Examples of good practices are:
- The reliance on purely descriptive statistics if inferential (ie, P-value generating) procedures would be underpowered
- The use of statistical procedures that are intended for small samples (eg, Fisher exact test)
- Emphasizing qualitative analyses and results for studies that have them
- Is the statistical test appropriate?
- Have the authors utilized the best procedures (not just appropriate), and have they controlled, either experimentally or statistically, for confounding factors?
- Is a comparison group appropriate, and if so, was a comparison group (of any sort) used? A comparison with self (eg, a time series), or a simple descriptive study, may not require a separate comparison group.
- Are the instruments employed valid and reliable? How do the authors know?
For qualitative studies, consider:
- Methods across qualitative studies can vary extensively. This is appropriate. Ideally, a qualitative study will describe a broad type of study (eg, grounded theory, phenomenological, content analysis, etc)
- There are basic components that should be present in all qualitative studies:
- Data collection procedures or sources
- Sampling methods
- Analytic processes
- Transcription procedures
- Some statements about control or consideration of biases
- A general sense that the authors have not selectively identified only quotes that fit their research question
- Additionally, the issues that apply to quantitative studies (described above) may be applied to qualitative study assessment, with appropriate consideration for how those requirements fit the qualitative model described.
1. US Department of Health and Human Services, Office for Human Research Protections. 45 CFR 46. https://www.hhs.gov/ohrp/regulations-and-policy/regulations/45-cfr-46/index.html#46.101. Accessed April 21, 2017.
2. The Kirkpatrick Partners. The Kirkpatrick Model. http://www.kirkpatrickpartners.com/Our-Philosophy/The-Kirkpatrick-Model. Accessed April 21, 2017.
Download a PDF of the Quality Guidelines