Contents
1. PREreview Open Reviewers
PREreview Open Reviewers is a mentoring program that empowers early-career researchers (ECRs) to contribute to scholarly peer review, centering around issues of equity, diversity, and inclusion.
Goals of the program
- Train ECRs on how to write constructive manuscript reviews, how to recognize and combat biases, and understand systems of oppression as they manifest within the peer review process.
- Pair ECRs with more experienced reviewers and journal editors who provide mentorship and constructive feedback to writing manuscript reviews.
- Help ECRs access a support network of peers and editors.
- Help bring more racial diversity to the peer reviewer pool—in the pilot we reserved 50% of the mentee slots to applicants who self-identified as underrepresented racial minorities as defined by the National Science Foundation (NSF).
- Work together with the mentors to build a database of trained reviewers that journal editors can easily access when selecting reviewers.
Structure of the program
The program is designed to support ECRs through a path of guided learning to build their profile as socially-conscious, constructive peer reviewers, and equip them with the skills to become mentors themselves. This framework is designed after the Mozilla Open Leaders' program and has the potential to impact hundreds of researchers over a relatively short amount of time.
We define four "Stages of Engagement", the first two of which are mentees, and the last two of which are mentors. The program can be entered by an individual at any stage depending on their experience with writing manuscript reviews (Figure 1).
To allow for different levels and paces of engagement, the program has a 14-week cohort-based component involving stage 1 and 4 participants, and an 8-month self-guided component involving stage 2 and 3 participants (Figure 2).
Stage 1 Review Trainees (RTs): These are researchers with little-to-no experience in peer review who want to learn more about peer review and its process in the live, cohort-based component of the program (see below). Upon completion, RTs will have published 2 preprint reviews on PREreview.org to start their open peer review profiles, and will be ready, should they want to, to join stage 2.
Stage 2 Advanced Review Trainees (ARTs): These are researchers who have completed 2-5 manuscript reviews in traditional journal peer-review settings or through open, community review of preprints (e.g., researchers who have completed stage 1), and want to learn more about peer review and its process in the self-guided component of the program (see below). Upon completion, ARTs will have published an additional 4 preprint reviews on PREreview.org to build on their open peer review profiles. With a total of 6 highly scored manuscript reviews, ARTs will be added to a Trained Reviewer list for editorial prospecting and will qualify to join stage 3.
Stage 3 Advanced Reviewer (ARs): These are researchers who have completed at least 6 manuscript reviews in traditional journal peer-review settings or through open, community review of preprints (e.g., researchers who have completed stage 2), and want to learn more about elements of implicit bias, how systems of oppression manifest in peer review, and build their mentoring skills by supporting ARTs in the self-guided component of the program. Upon completion, will be added to a Trained Reviewer list for editorial prospecting (if they haven't already via stage 2), and will qualify to join stage 4.
Stage 4 Review Mentors (RMs): These are experienced reviewers who have either completed the training in stage 3 or enter the program with extensive experience in peer review (e.g., journal editors). They are researchers who want to learn more about elements of implicit bias, how systems of oppression manifest in peer review, and are willing to mentor RTs in the cohort-based component of the program.
Cohort-based component
During the 14-week cohort-based component of the program, RTs engage in community calls, peer-to-peer support chats, and one-to-one RTs/RMs calls. The curriculum focuses on the critical evaluation of emerging scientific literature, strategies to write constructive reviews, understanding how systems of oppression manifest within the scientific enterprise, and helping mentees recognize and challenge their own internal biases that may influence their ability to remain objective when reviewing (Figure 3). In addition, we provide opportunities for peer-to-peer support, and mentorship and leadership training for those who wish to be mentors in future cohorts, an effort that will help make the program grow in its impact and reach.
Self-guided component
During the 8-month self-guided component of the program (Figure 4), ARTs have access to training materials developed for the cohort-based component in the form of recorded videos, documents, and other resources that they can consume at their own pace. ARTs are paired with ARs and meet for 30-60 minute calls once a month. With the support and guidance of their mentors ARs, ARTs complete and publish 4 preprint reviews on PREreview.org. Prior to their first one-to-one call, ARs watch our recorded Mentor Thought Exercise video to encourage the development of a supportive, nurturing, and safe space for ARTs to develop their peer reviewing skills.
A note about our pilot
The PREreview Open Reviewers pilot launched in October 2020 with the cohort-based component of the program. Applications to join stage 1, 2, 3 were open for 2 weeks in September 2020 and shared with a limited number of people in our network. We received a total of 51 ECRs applications! For our pilot, we selected 10 applicants to enter stage 1, 8 applicants to enter stage 2 and 3, and identified 9 journal editors to enter stage 4. We hope to expand the number of ECRs in future cohorts as a number of non-selected applicants expressed a high desire to participate, demonstrating the need for such a program within the scientific community. Half of the selected applicants self-identified as underrepresented minorities (see below for what we mean by that). Stage 4 RMs are experienced journal editors who support our mission and are volunteering their time to train stage 1 RTs. Read more about our pilot participants here.
Our long-term goal with this program is to help address racial and global underrepresentation in peer review. Due to time zone and staff constraints, our pilot program mainly accommodates U.S. and European applicants and uses the National Science Foundation’s definition of underrepresented racial minorities in science, which includes Black/African Americans, LatinX, Indigenous/First Peoples/American Indians & Alaskan Natives. Thus, the designation of underrepresented minorities is defined using U.S. statistics rather than global statistics. In upcoming years, as funding and support for this project hopefully increase, we aim to expand our definition of underrepresented minorities in research to become more globally inclusive and accessible.
As researchers who have trained in North America and have degrees in the life sciences, we also recognize our inherent cultural and scientific bias in how we developed this program. As our program continues to grow, especially as we expand it globally through partnerships with like-minded organizations that share our values, we will adapt our curriculum and training to address these biases. We understand and are committed to the necessity of continual reassessment in order to avoid recapitulating systems of oppression within our own programming.
This pilot was possible thanks to the generous support of eLife and Mozilla.
Contact
To learn more about this program, give us constructive feedback, share ideas on how to improve it moving forward, please don't hesitate to contact us at mentoring@prereview.org.
2. Live-streamed preprint journal clubs
Live-streamed preprint journal clubs are topic-centered, interactive preprint journal clubs that are live-streamed via a video-conference tool such as Zoom.
Core to PREreview's mission is increasing diversity and inclusion in scholarly peer review. The live-streamed preprint journal clubs allow structured and constructive discussions following guided discussion questions that encourage diverse methods of participation. Through these events, we strive to support and empower the voices of our fellow researchers who have historically been underrepresented in scholarship and who are in the beginning stages of their research careers.
Who participates?
- Two facilitators from our team with experience in hosting live preprint journal clubs
- A topic expert (optional) to guide participants through a constructive discussion
- Preprint authors (optional) to gather live feedback on their preprint
- Researchers who join from all over the world to collaborate on improving a preprint, build their network, and meet globally-renowned experts
What are the event outputs?
After the call, our team works with the journal club participants to summarize the discussion into a peer review report that is openly shared on PREreview and can even be integrated into the journal-organized peer review process—e.g., a PREreview included as reviewer #3 in the eLife peer review process on a bioRxiv preprint.
Who has joined us before?
We have previously collaborated with PLOS, who partnered with us to run three such events during OAWeek in 2018, JMIR Publications who partnered with us to run a COVID-19 focused virtual journal club on April 14, and eLife who partnered with us to run a 5-series preprint journal club from June through October 2020.
Ready for your own preprint journal club?
To request our support to run a live-streamed preprint journal club, you can email us at contact@prereview.org.
Acknowledgements
We thank the Mozilla Foundation for sponsoring the PREreview Open Reviewers pilot. We also thank the Mozilla Open Leaders community, in particular Abigail Cabunoc-Mayes and Chad Sansing for pioneering a completely online and interactive program that empowered people to be leaders in their communities from all over the world, and that provided us with the skills and inspiration to build our own peer review mentoring program.
We also thank eLife for sponsoring the PREreview Open Reviewers pilot by directly supporting our Program Manager, Dr. Antoinette Foster. Read more about their support here.
We thank Drs. Emmy Tsang, Yo Yehudi, and Julia Lawndes for advising us on how to frame and organize a spin-off of the Mozilla Open Leaders, adapting it to our needs and those of the communities we want to serve.
We thank Dr. Andre Walcott for designing and co-facilitating the Mentor Thought Exercise.
We thank Maria Guerreiro, Journal Development Editor at eLife, for helping us connect with the eLife Editors who serve as stage 4 RMs in our pilot.
We thank Dr. Veronique Kiermer, Chief Scientific Officer at PLOS and member of the PREreview Steering Committee, for her advice and help connecting our team with PLOS Editors.
We thank Lindsay Morton, Senior Manager Contributor Outreach at PLOS, and Dan Morgan, Director of Communications & Community Relations at PLOS, for being our first guest speakers and sharing their knowledge with our RTs on how to be effective and constructive reviewers.