At PREreview, we believe all researchers should be able to engage in honest and respectful conversations around emerging scientific output. Peer review is a fundamental aspect of being a researcher. However, formal peer review training is uncommon and varies in quality.
Our hope is that by engaging in open peer review of preprints, possibly collaboratively as a result of a discussion at a journal club, researchers will have the chance to practice peer review in the safe and welcoming environment provided by PREreview, and learn from other openly shared PREreviews and comments.
5 golden rules for a great PREreview:
- Be respectful. We are all human beings and we all make mistakes. The goal here is to help each other move knowledge and discovery forward and not make your peers feel diminished or personally attacked by disrespectful comments.
- Be constructive. Similarly, you need to make sure feedback stays constructive and actionable so that preprint authors can easily respond to the feedback and possibly incorporate the constructive suggestions into the final publication.
- Be honest. Being constructive does not mean you have to lie or only bring up positive comments. It means you have to write your suggestions in a way that is not insulting to the authors and that can lead to easy incorporation of that feedback. Constructive negative comments followed by rational and possibly examples/suggestions on how to improve the issue are welcome.
- Be clear. The clearer you are about what your comment or suggestion is referring to, the easier it will be for authors and readers to understand it.
- Be concise. You can be as thorough as you have time to be, but keeping your feedback concise increases the chances authors and other readers will read through it and take action as a result of it.
More on giving constructive feedback
Giving feedback is hard. Doing so constructively can be even more challenging. But what does "constructive" mean in the context of PREreview? Does it mean that you can only express positive comments related to a study? No.
With constructive feedback, we refer to feedback that can be used by the authors to improve their manuscript. That includes providing negative feedback on the use of a statistical test for example, or on the interpretability of a particular bar plot. However, that same negative feedback has to be provided using respectful language and in a way that allows the authors to fix the problem.
For example, if you believe a statistical test was used incorrectly, you may say:
"Statistical test [test1 name] should be used only if the data is distributed normally. The data presented in this preprint appear to be highly skewed to the left, and for that, I would suggest the use of [test2 name] test, a non-parametric version of [test name1] which makes no assumption on the parameters of the distribution of the data. If the authors' choice of [test1 name] is motivated by a particular strategy or other non-obvious analytical constraints, I recommend the authors explicitly mention that in the Methods section justifying their choice accordingly."
This is constructive!
As opposed to:
"You have no idea of what they are doing and should go back to statistics 101."
This is a clear violation of the PREreview Code of Conduct.
In the first version of the comment, you are providing your opinion and backing it up with a reason why you think that test is inappropriate. Furthermore, you are using "the authors ..." as the subject of your sentences instead of "you ..." which helps depersonalize the feedback and keep the focus on the work.
In the second version of the comment, you are clearly attacking the author at a personal level and insulting their education, which is not only offensive, unprofessional, and a clear violation of the Code of Conduct, but it is also useless to the authors as it does not provide a way to improve the study.
Remember, you can be honest and constructive at the same time!
As our friends at PLOS say, try to follow the golden rule:
"Write the type of review you’d want to receive if you were the author."
For additional precious tips on how to provide constructive feedback, explore the PLOS Peer Reviewer Center.
Being on the receiving end of feedback
We established that giving feedback can be challenging, but it can (and should) be done constructively following some tips. What about receiving feedback? The world often does not prepare us to receive feedback either. Often even if feedback is given constructively, we tend to take it personally and assume that the person who gave us feedback is deliberately challenging our identity as people.
If that is your first reaction to a comment, read it again and try to imagine how you would have given that same suggestion to your peer if you had any reason to believe, like in the example above, a statistical test was not used correctly.
If after considering the suggestion(s) made by the PREreviewer(s) you still believe the way you and your team analyzed the data is correct, provide an explanation of why. You can also consider asking the PREreviewer for further explanation around the suggestion, with examples and evidence to back it up.
Remember that PREreviewers are community members who are here to help you improve your manuscript and provide more transparency to the evaluation process of scholarly outputs – in this case, of preprints. They are finding time in their busy schedules to help you. Be polite, respectful, and grateful for their contribution.
We invite you to reply to the PREreview with a comment, thanking the PREreviewer(s) for their time. If you found the PREreview particularly helpful, give it a Plaudit and consider acknowledging the PREreviewer(s) in a new version of the preprint and/or the final version of the manuscript issued for publication.
If you believe the content of a PREreview or a comment violates our Code of Conduct, please report the incident immediately by emailing us at firstname.lastname@example.org and we will do our best to act according to our policy.
In order to help you write a PREreview, we have two basic templates you can load directly into the editing window when you write your PREreview. Below is the content of these two templates.
Template 1 is a quick questionnaire you can copy and paste in a collaborative document during journal club, for example, or load to guide you in the review process.
Template 2 is a guide for a more detailed report you might want to write, more similar to what you would be asked to write if you were selected as a reviewer for a journal.
We are keen to update them based on your feedback, so please don't hesitate to contact us with any suggestions or questions at email@example.com.
Below is a list of quick questions and suggestions you can use to guide your reading and reviewing of the selected preprint to produce a quick review.
- What is the main question the study attempts to answer?
- What is (are) the hypothesis(es)?
- What techniques/analyses do the researchers adopt to test their hypothesis(es)?
- Why is this study relevant?
- Write here any general comments you might have about the research approach.
- Write here any specific comment you might have about experimental approaches and methods used in the study.
- Write here any specific comment/note about figures in the preprint (this could be related to the way data are displayed and your ability to understand the results just by looking at the figures).
- Write here any additional comment you might have (this includes minor concerns such as typos and structure of the manuscript).
Below is a detailed guide to help you critically and constructively evaluate the selected preprint.
Overview and take-home message
Write a short (1 paragraph) summary of what the main findings of the research were and how this work has moved the field forward. This could take the following structure:
“ [First author] et al., have made significant advances in [x, y, and z] by showing that… In addition, they have bridged the gap in our knowledge about how [process i interacts with process ii]. Although this work is of [significant] interest in the field, there are some concerns that could be addressed in the next version. These are outlined below.”
As the aim of this PREreview is to support the authors by providing constructive feedback, the PREreviewer(s) should also include positive remarks to encourage future posting of preprints. Remember, the authors are human too!
To help guide you, here are a few questions you might ask yourself after reading the preprint:
- What results did you find the most interesting and why?
- What did you learn from the study that really fascinated you?
- Did you learn anything from this preprint that you would include in your future work? Examples could be a new technique, a new way of approaching an idea, a certain way of interpreting the type of data, a good way of summarizing the results, the structure of the manuscript if it is clear and logical, etc.
- What ideas for further research are based on the preprint? [Note: this encourages preprint authors for their early sharing and promotes openness among reviewers as well.]
Next are your concerns. Remember you are writing to another human being, so be respectful and realistic. Also, if you are unsure about something, be honest and say you are unsure. Don’t feel afraid to admit if you are not confident about your concerns, or part of the work is out of your area of expertise.
Here the reviewer(s) should list 2-3 major concerns about the research (if they exist). This may relate to:
- Does the title accurately reflect the results?
- Experimental design: were the techniques appropriate for the hypothesis? Were suitable controls included? Were new techniques/mutants/cell lines etc. sufficiently characterized to allow the study to be replicated? Were robust statistical methods used?
- Do the results support the authors’ conclusions and, if not, what additional experiments would you suggest?
- Are there big flaws in the understanding of the current literature and interpretation of the literature that might undermine the interpretation of the current results?
Here the PREreviewer(s) can mention 3-5 minor concerns that are not critical to the understanding and conclusions of the research but would improve the overall flow or clarity of the manuscript. These concerns might include the following questions:
- In general, was the manuscript easy to follow/did it have a logical flow? Were there a lot of typos that needed addressing?
- Does the abstract provide a concise summary of the hypotheses and main results?
- Were the methods sufficiently detailed to allow the experiments to be repeated?
- Were the figures clear and logically arranged? Were the figure legends sufficiently detailed to allow the figures to be understood without reading the main text?
- Are the figures representative of the data described in the results section?
- Did the authors make all their associated data openly available (e.g. sequencing data, coding)?
- Did the discussion address how their results move the field forwards?
- Were the citations thorough or were key references missing?
You can find additional peer review tips in the PLOS Peer Reviewer Center, including their How To Peer Review page that contains 10 Tips for getting started as a reviewer. These resources include checklists, videos, and further information about ethical considerations etc. They also provide suggested language you can use to address common issues that can arise when reviewing a manuscript.
How to start a preprint journal club
10 simple tips – also available as slides here
1. Find your community
The opportunist: If you attend a regular (post-publication) journal club (JC), you can start by presenting a preprint whenever it is your turn to present.
The revolutionist: If you are not currently attending a JC but there are JCs going on at your institution, you can start by joining one (JCs are fun and a useful way to stay up-to-date with scientific topics of interest) and then present a preprint on the relevant topic when it is your turn to present.
The pioneer: If you are not currently attending a JC and there aren't JCs that interest you, start your own! You can try to get support from your program or department and ask for a bit of funding to fuel the JC with coffee and donuts (this will help convince researchers to attend, particularly if the JC is in the morning). Often, you can seek the help of the library, the department chair, the program director, or faculty. It is also possible to organize a preprint JC class for which students can get credit (see Prachee Avasthi's syllabus). The advantage of offering a preprint JC as a course is that you would help formalize training for early-career researchers on how to so peer review. It is worth considering though that by labeling it as a course, you might only get students to attend.
Whichever strategy you use to find your community, when you start your first preprint JC spend 2-3 minutes at the beginning of your presentation to explain what preprints are (many researchers already know that but some students might not), why you think it's important to read and discuss preprints at JCs, and the benefits of writing up feedback for everyone to see.
2. Find a preprint to review
Where to look: There are many preprint servers that differ in several dimensions, the first of which are field and geographical location. To find a preprint server in your field you can look in Research Preprint Servers List. This can help you find a preprint that interests you.
Get notified: Some preprint servers (e.g., bioRxiv) have a way to set up notifications using keywords that will send you an email whenever a new preprint with that particular keyword is posted on the server.
Stay connected: Many researchers these days use Twitter and other social media platforms to share preprints they found interesting to read or to publicize their own. Follow the hashtag #preprint and the preprint servers' handle (@) to get them in your feed. There are also bots tweeting out preprints with the highest visibility on a given preprint server (e.g., Promising Preprints), sites that rank preprints based on article download metrics or Twitter attention (e.g., Rxivist), or search engines that help highlight preprints that for one reason or another have remained in the shade (e.g., the newly developed Hidden Gems search). It is also useful to follow other researchers in your field who might share content that you may find interesting.
3. Preparation is key
Use a collaborative doc to take notes: In preparation for your JC, prepare in advance a document in which you or someone else from the group can take notes on the main discussion points. The collaborative document could be a Google Doc or Etherpad. You can also use Template 1 (see above) to guide the discussion with some pre-written questions. You can then share the document ahead of time to allow any eager participants to fill out the questions with notes before the meeting. This step is important because by taking notes you will have a much easier time to compile the feedback to share with the community (e.g., on PREreview!).
4. Work in pairs
Host + Scribe = Winning team!: If you are the presenter of the preprint, it might be difficult for you to both lead the discussion and take notes. For this reason, it is advisable to find another person in the group who is willing to annotate the discussion in the collaborative document.
5. Keep it constructive
Set the right tone: Arguably the most important of all tips. We cannot emphasize enough how important it is to set the right tone for the discussion to ensure everyone feels safe and welcome to participate. Encourage respectful and constructive comments. As we mention above, being constructive does not necessarily equal limiting comments to positive ones.
Remind JC participants that being constructive means providing feedback that is useful to the authors and the PREreview's reader. Encourage actionable and possibly detailed feedback, rich in examples and perhaps even suggestions on how to do something differently if found to be needed. Note that one section of our CoC is dedicated to the preprint JC. We encourage you to read this aloud or summarize the main points to the JC participants.
6. Write the PREreview
Identify co-authors: Would all participants of the JC discussion like to be listed as co-authors of the PREreview? If so, you can invite them to sign up to PREreview (they will need an ORCID iD) or you can list their names (ask for consent) at the top of the PREreview as participants of a JC discussion. If they are interested in being co-authors, ask for support in writing the full report, or perhaps just editing it as the more people that can help, the easier it will be to write the PREreview.
Choose a template: On the new PREreview platform, you have the option to choose one of two available templates. The first one is a quick questionnaire that will guide you and other JC participants through a constructive discussion of the preprint. The second one is a longer and more detailed guide that will help you craft a complete report similar to what you would do if you were peer reviewing for a journal.
Keep it constructive and simple: As we said in tip 5, keep your feedback constructive and actionable, so that it is simple for the authors to integrate into the manuscript. Also remember that any content on PREreview that violates the Code of Conduct will be removed from the site, and the author(s) of those comments may be permanently banned from the community. Check out the Code of Conduct during JC discussions.
7. Share your PREreview
Once the PREreview is written, you can share it openly on PREreview and get a free digital object identifier (DOI) associated with it. This will allow you and other PREreviewers to share your feedback openly and list your contribution to peer review in your CV, resume, or personal webpage. Another option is to share the feedback with the preprint authors privately via email, but if you do this no one else will be able to see your contribution and learn from your feedback.
8. Practice makes it perfect
Ask for feedback on your PREreview: In the next version of the PREreview platform there will be a button to allow you to share the content of a PREreview with another person who you might want to review/edit your work before making it public – this can be your mentor for example. This option can be particularly helpful if one of your goals is to learn how to peer review and you have a mentor who is willing to help you in the process. In the meantime, you can copy and paste your review in a Google doc for example and have a trusted peer review it before publishing it on PREreview.
Review regularly: You will quickly realize that the more you discuss and review preprints the easier it becomes to provide constructive feedback and to write up a PREreview.
9. Liven it up and build your network
Lead and participate in live-streamed PREreview journal clubs: Live-streamed preprint journal clubs, are journal clubs that happen online and involve researchers from different institutions and geographical locations who have in common a passion for a given topic. These events are not only more inclusive because they allow anyone with an internet connection to participate, but they also help you build connections with your peers. Below you can learn more about them and how we can support you if you wish to get one going below.
10. Look for more resources
Check out our email templates to help you get started with the JC in your department or our tips on how to lead a live-streamed preprint JC (see below).
The PLOS Peer Reviewer Center has many, many resources to help guide you through the peer review process with dos and don'ts that are very useful when applied to any feedback you may be giving to your peers.
Additional tips for creating a safe space
To encourage a respectful and thriving preprint JC environment, we have additional tips we strongly recommend you follow for hosting and participating in local and online (live-streamed) preprint JCs.
Points to consider when organizing a preprint JC
Before each journal club meets, allow an anonymous vote in which all participants have the opportunity to express any discomfort or conflict of interest related to reviewing a particular preprint. One vote against the preprint is sufficient to veto that preprint for discussion at the JC. This will help to create a safe environment for all your JC participants.
Note-taking should be encouraged for all participants. However, we recommend that you assign one primary note-taker for each JC and have a system to rotate this responsibility across all JC members so that stereotypes, for example, that women should take notes, do not permeate this process.
Responsibility for hosting the preprint JC should rotate between all members of the JC, and this should be scheduled well in advance to allow for preparation.
All contributions to the PREreview should be acknowledged, if desired by the contributor, within the PREreview. What constitutes authorship should be pre-determined prior to compiling the review.
Points to consider when participating in a preprint JC
- Be respectful of other participant's contributions and do not dismiss their comments in an aggressive manner.
- Acknowledge and amplify constructive comments that have been overlooked.
- Be aware when you or others are dominating the discussion and allow space for other participants to contribute.
Email templates to get started
Below, we have written three template emails you can tweak and send to your department to invite students, postdocs, and faculty to join the Preprint JC.
Template email #1 (long and informational): Invite your colleagues to join you
Dear students, postdocs, and faculty,
I am … [Introduce yourself if you are not already known by everyone]. I would like to start a preprint journal club (JC) in our department. I recently decided to contribute to PREreview’s [optional: add link] effort to promote the use and dissemination of preprints in academia and their review. For those of you who are not familiar with them, preprints are complete pieces of scientific work that have not yet undergone editorial peer review.
The main benefits of posting preprints are that i) you, the author(s), can immediately share your work with the scientific community (but really with everyone, since preprints are open access, i.e., free!) without having to wait for the publisher’s timeline, and ii) the community learn about your research early and can comment on the manuscript, potentially improving your work – since preprints are dynamic articles, you can incorporate the feedback and upload revisions to the preprint server of your choice.
Discussing preprints at JC will be beneficial to us as we will get to know the advancement in our field 6 months to a year earlier than we would if we waited for the publication. At the end of each JC, we will compile our comments into a review and post it as a public document on PREreview.org, a web platform that hosts preprint reviews and assigns them DOIs for free. In addition to this being useful to the authors and others who are interested in the science, I see posting reviews as an opportunity for early career researchers like myself to learn how to write a peer review.
[Here you can either invite people to join the first meeting you have already scheduled] Our first meeting will be on … at the .... Join us and let’s learn together!
[...or you can ask for help organizing the JC and brainstorm for ideas on when, where, and how to do it to maximize attendance in your department, etc.]
If you want to help me organize the preprint JC, please reply to this email. Ideally, I would like to meet for JC weekly or every other week, at a time and day of the week that best work for everybody who is interested.
Template email #2: Short(er) and to the point
Dear students, postdocs, and faculty,
I am … [Introduce yourself if you are not already known by everyone]. I would like to start a preprint journal club (JC) in our department. Preprints are complete pieces of scientific work that have not yet undergone editorial peer review. At the end of each JC, we will compile our comments into a review and post it as a public document on PREreview.org, a web platform that hosts preprint reviews and assigns them DOIs for free.
[Here you can either invite people to join the first meeting you have already scheduled] Our first meeting will be on … at the .... Join us and let’s learn together! [...or you can ask for help organizing the JC and brainstorm for ideas on when, where, and how to do it to maximize attendance in your department, etc.]
If you want to help me organize the Preprint JC, please reply to this email. Ideally, I would like to meet for JC weekly or every other week, at a time and day of the week that best work for everybody who is interested.
Template email #3: Q&A format—good for generating initial interest, more informal
(This template was written by Dr. Emmy Tsang, now Innovation Officer at eLIFE)
I'm running a preprint Journal Club. 30-second summary: if you have an hour to spare to discuss latest science, help fellow researchers, make a difference to publishing, and eat [food], [put your name here/reply to this email]
What is a preprint? A manuscript that has yet undergone journal-organized peer review.
What is a Journal Club? An occasion where we sit together to discuss a scientific paper—what we like and don't like What is SPECIAL about a preprint journal club? We discuss a preprint instead of a published paper. Because the preprint is not yet published, our comments and suggestions might actually make a difference to the final published paper.
Cool... but how is that done? Via PREreview.org, we will publish our feedback as a review for the preprint. This PRE-review gets a DOI, will be public, and will also be sent to the authors of the preprint.
Why should I spend time and effort to give feedback to other researchers' manuscripts?
1. if you're a student: it's a really important part of your scientific career, to be able to understand and critique other people's research and results.
2. postdocs + students: reviewing papers is a huge part of a PI's job- this is an excellent opportunity to practice 3. everyone: publishing sucks. There's a lot of talk about how long and tedious the process is, but here's a chance to actually do something to change it. You're contributing to the exploration of alternative ways to get results out and advance science faster.
"But... I'm just a student, I don't know how to write reviews/make valid scientific comments..." Well, I don't either. That's why this is a discussion—we can teach each other! There will be some materials to guide us through (also so that it doesn't go on forever!).
I have tons of experiments. I don't have time. Free food, and I promise it will finish on time!
"OK, I'm sold. What do I do next?" Reply to this email, so I can keep you in the loop! My plan is [some specifics about the JC. You can also ask for suggestions on the format, preprints, etc.]
"I have questions/suggestions..." come talk to me! This is honestly an experiment- I would love to hear from you how to make it work :)
Template email #4: send it to the preprint's corresponding author after you publish your review
Dear [corresponding author's name],
I am [insert your name and affiliation] and together with other researchers, I recently discussed your preprint ([insert title or DOI of preprint]) during our journal club meeting. I am a member of PREreview, a platform and a community that aims to support early-career researchers in presenting and reviewing preprints by providing resources to guide them through the process of peer review. Using these resources, we have compiled a review of your preprint, summarizing the feedback from the journal club. The review is openly available on PREreview.org ([insert DOI or the link to your review]).
We hope the preprint review will be useful to you, and we welcome any comments you may have in response. If you would like to respond to the review, you can leave comments directly on the review by selecting the text and clicking on the commenting icon at the top of the page.
How to start and lead a Live-Streamed PREreview JC
Live-streamed preprint journal clubs (#LivePREJC) are topic-centered, interactive preprint journal clubs that are live-streamed via video-conference.
These events are designed to be inclusive by allowing structured and constructive discussions around preprints and encouraging diverse methods of participation. Researchers—or anyone interested in joining the discussion—from all over the world can join to build their network, meet globally-renowned experts, and collaborate on improving a preprint. Previous live-streamed preprint JCs we have organized attracted participation from researchers, students, publishers and artists!
You can request our help to organize a live-streamed PREreview journal club by filling this form. Live-streamed PREreview journal clubs are:
- Inclusive: anyone, anywhere in the world with an internet or phone connection can join
- Informative: you can learn more about the topic by listening to/reading the comments of other researchers in the field and even the authors themselves (if invited)
- Efficient: if preprint authors are present, they receive feedback in real-time. Also, we limit the calls to 1 hour to keep the feedback focused and the calls on time
- Collaborative: the format encourages contributions from all participants regardless of participation style preference, i.e., both vocal and silent writing (etherpadding)
- Fun: even though the discussions are kept professional and centered around providing constructive feedback to the preprint, they are a fun way to meet other people interested in the field.
To get started
- Choose a preprint you wish to discuss at a live-streamed preprint JC.
- If you can, make a list of a few other scientists or other interested professionals, preferably at different career levels and from different institutions, who you think would be interested in participating in the discussion of the chosen preprint. If you can't think of anyone to invite, don't despair... our team will help you recruit more participants.
- Fill out this form to request our support to run the live-streamed preprint JC.
What to expect
Our team will help you organize the call by providing a registration form, a customized collaborative note-taking document (example template from a previous live-streamed preprint JC here), and access to a Zoom virtual room (up to 100 participants). We will also send out an email to participants who you have identified or who have signed up to participate with instructions on how to join the live-streamed preprint JC. Participants will be encouraged to read the preprint before the event, to keep the discussion short and focused.
Should you desire, we will help you recruit participants by inviting people in our network and advertising the event on social media using the #LivePREJC hashtag—and any other hashtags related to the research field of the preprint discussed.
One or two members of our team will facilitate the call to ensure that the conversation runs smoothly and stays on time (1 hour). During the call, we will take notes and encourage others to take notes on a collaborative Google Doc or Etherpad. Importantly, we will set the tone for a productive and respectful conversation according to our Code of Conduct.
If you are a preprint author, you will have the option to participate in the discussion, you will be asked to please remain in ‘listening mode’—except when asked a question by participants—until the last 10 minutes reserved for this discussion. This will encourage participants to express their constructive feedback freely and stimulate a productive discussion.
If you are NOT a preprint author, we will invite you to contact the corresponding author(s) and let them know about the upcoming live-streamed preprint JC. If you don't feel comfortable doing it, please let us know and we will do it on your behalf. Letting the preprint authors know that there will be a live-streamed discussion about their preprint is not necessary, but it is recommended. You can also choose to invite them to join the call.
Should you desire to, we will help you format and write the PREreview, and share it on the PREreview platform.
Questions? Feedback? Want to help improve these resources? Please contact our team at firstname.lastname@example.org.