Peer review, just as any other process, protocol, policy, structure, is conducted by people and thus is naturally subject to influence by the biases of people. In rethinking a more open and equitable way of evaluating research, it is imperative to reflect on past and present mistakes that have led to the inequitable system of today, and to engage the research community in designing and implementing that change.
PREreview Open Reviewers is a training and mentoring program and workshop that empowers early-career researchers (ECRs) to contribute to scholarly peer review, centering around issues of equity, diversity, and inclusion.
Open Reviewers is offered as three-part workshop or as a 14-week to 8-month long program. Below is information about these two formats. Both formats are free to participants.
The foundation of the Open Reviewers coursework includes a Reviewer Guide, a Bias Reflection Guide, and a Review Assessment Rubric that can be found on Zenodo. Find out more about these free resources here.
Open Reviewers Workshop
What is it and who is it for?
The Open Reviewers Workshop is a condensed version of the Open Reviewers Program (see below) designed to train researchers across career levels and areas of expertise in how to engage in manuscript peer review, centering around issues of equity, diversity, and inclusion. It is meant to be delivered to groups of researchers belonging to the same research group, institution, or university (here referred to as the “host organization”).
- To know how to give clear, constructive, and actionable feedback
- To know how systems of oppression manifest in the peer-review process
- To know how to identify and address biases in peer review
- To know how to write a manuscript review for a journal or self-organized to a preprint
- To know how to engage in the open peer review of preprints and build a public profile as a reviewer
The Open Reviewers Workshop has three basic modules facilitated by our team. We recommend the three modules take place over the course of a five-week period, with one module every other week. A fourth module can be added to the workshop to engage participants in individualized mentorship opportunities that extend for a period of up to one month after the end of the workshop.
The workshop is open to any audience the host organization may select and is designed to engage participants in hands-on learning activities and small-group reflections.
The optimal number of participants is 30. However, a larger group of participants can be accommodated by adapting the format and delivery of the content to maximize the learning experience for everyone.
The workshop’s content per module is outlined below:
- Introduction to the Open Reviewers Workshop
- Methods of peer review: how it’s done and what can be improved
- Systems of oppression in peer review: what they are and why it’s important to recognize and combat them
- ASSIGNMENT: Applying an equity lens to addressing issues in peer review
- Reviewer Guide walk-through: Six steps to writing a peer review
- Giving clear, constructive, actionable feedback
- Assessing and addressing biases in peer review
- Elements of non-violent communication and conflict resolution
- ASSIGNMENT: Read preprint in preparation for Module 3
- Guided live and collaborative review of a preprint
- OUTPUTS: Publication of the collaborative preprint review authored by participants made openly available under CC-BY 4.0 on PREreview.org (DOI assigned via Zenodo); Badge of participation for participants who are registered on PREreview.org; Certificate of participation
Module 4 (optional)
- Participants with little-to-no experience in peer review (mentees) are paired with mentors from the same research institution to receive individualized feedback on a preprint review
- Mentors need to have attended Module 1 and 2 in order to qualify
- OUTPUTS: Publication of the preprint review authored by mentees made openly available under CC-BY 4.0 on PREreview.org (DOI assigned via Zenodo); Mentor role recognition on the preprint publication; Badge for mentors who are registered on PREreview.org
PREreview deliverables for host organizations will include:
- Customizable templates for outreach and registration materials to recruit participants
- Up to 5 hours, 1:1 consultation with host organization’s organizers to adapt resources to the specific needs of their targeted audience
- Facilitation of three modules, delivered virtually via Zoom or any other preferred video conferencing tool
- Close-caption services via Otter.ai, with live-caption and/or American Sign Language interpretation available upon request
- Communication with participants throughout the duration of the workshop
- Workshop impact assessment and individualized participants’ learning dashboard via short pre- and post-workshop surveys
- All resources made available to the host organization and workshop participants under CC-BY 4.0, including transcripts, audio and/or video recordings if applicable
- Posting of the full PREreview as result of Module 3 collaborative review session with post-workshop engagement of participants who opt in authoring the public review
- OPTIONAL if Module 4 is included: Support in recruiting and pairing mentors and mentees, with delivery of extra resources and materials to guide individualized mentorship (including guidelines for mentors, and clear expectations for all parties involved); Support in posting the full PREreviews as result of Module 4 individual reviews
Language and cultural context
The content of the workshop and the language of delivery is English. Automated closed-caption services and transcripts are provided via Otter.ai. Speech-to-text live-caption and/or American Sign Language interpretation services are available upon request.
The content was developed by a group of researchers trained in North America and therefore uses terminologies and cultural references that are consonant to North American culture.
Our long-term goal is to grow our impact and work with global audiences. To this goal, we work with other organizations to adapt the content of the workshop to local needs and contexts, and help train and empower local leaders in delivering the workshop to their communities.
The total cost of the three-part workshop for a maximum of 30 participants excluding organizers is $5,000 USD to be paid in advance to Code for Science & Society, Inc, the non-profit fiscal sponsor of PREreview.
The additional Module 4 can be added for an extra $500 USD.
More than 30 participants can be included, but we will charge an extra $200/10 participants that will help us make the appropriate changes to content delivery and format to maximize the learning experience for everyone.
We offer a standard contract agreement, but can accommodate format requirements for the host organization as needed.
We understand that cost can be a barrier for underresourced host organizations. We are working on a transparent and equitable pricing tier for our workshops. In the meantime, if you are interested in hosting this workshop for your community, but you do not have sufficient funds, please contact us.
Please contact us for additional information at firstname.lastname@example.org.
Open Reviewers Program
In the fall of 2020, we piloted the Open Reviewers Program, a cohort-based mentoring and training program that empowers early-career researchers (ECRs, undergraduate students to junior faculty level) to contribute to scholarly peer review, centering around issues of equity, diversity, and inclusion. The pilot engaged 14 mentees and 13 mentors, and has now ended (read more about what we learned here.
The program was free to all participants and was possible because of the generous support by eLife and the Mozilla Foundation, as well as the mentors who volunteer their time to help their mentees develop and refine their peer review skills.
Below is a description of the program in its piloted form. Our team is fundraising to run our next cohort in 2022. Information about the next cohort will be available soon.
Goals of the program
- Train ECRs on how to write constructive manuscript reviews, how to recognize and combat biases, and understand systems of oppression as they manifest within the peer review process.
- Pair ECRs with more experienced reviewers and journal editors who provide mentorship and constructive feedback to writing manuscript reviews.
- Help ECRs access a support network of peers and editors.
- Help bring more racial diversity to the peer reviewer pool—in the pilot we reserved 50% of the mentee slots to applicants who self-identified as underrepresented racial minorities as defined by the National Science Foundation (NSF).
- Work together with the mentors to build a database of trained reviewers that journal editors can easily access when selecting reviewers.
Structure of the program
The program is designed to support ECRs through a path of guided learning to build their profile as socially-conscious, constructive peer reviewers, and equip them with the skills to become mentors themselves. This framework is designed after the Mozilla Open Leaders' program and has the potential to impact hundreds of researchers over a relatively short amount of time.
We define four "Stages of Engagement", the first two of which are mentees, and the last two of which are mentors. The program can be entered by an individual at any stage depending on their experience with writing manuscript reviews (Figure 1).
To allow for different levels and paces of engagement, the program has a 14-week cohort-based component involving stage 1 and 4 participants, and an 8-month self-guided component involving stage 2 and 3 participants (Figure 2).
Stage 1 Review Trainees (RTs): These are researchers with little-to-no experience in peer review who want to learn more about peer review and its process in the live, cohort-based component of the program (see below). Upon completion, RTs will have published 2 preprint reviews on PREreview.org to start their open peer review profiles, and will be ready, should they want to, to join stage 2.
Stage 2 Advanced Review Trainees (ARTs): These are researchers who have completed 2-5 manuscript reviews in traditional journal peer-review settings or through open, community review of preprints (e.g., researchers who have completed stage 1), and want to learn more about peer review and its process in the self-guided component of the program (see below). Upon completion, ARTs will have published an additional 4 preprint reviews on PREreview.org to build on their open peer review profiles. With a total of 6 highly scored manuscript reviews, ARTs will be added to a Trained Reviewer list for editorial prospecting and will qualify to join stage 3.
Stage 3 Advanced Reviewer (ARs): These are researchers who have completed at least 6 manuscript reviews in traditional journal peer-review settings or through open, community review of preprints (e.g., researchers who have completed stage 2), and want to learn more about elements of implicit bias, how systems of oppression manifest in peer review, and build their mentoring skills by supporting ARTs in the self-guided component of the program. Upon completion, will be added to a Trained Reviewer list for editorial prospecting (if they haven't already via stage 2), and will qualify to join stage 4.
Stage 4 Review Mentors (RMs): These are experienced reviewers who have either completed the training in stage 3 or enter the program with extensive experience in peer review (e.g., journal editors). They are researchers who want to learn more about elements of implicit bias, how systems of oppression manifest in peer review, and are willing to mentor RTs in the cohort-based component of the program.
During the 14-week cohort-based component of the program, RTs engage in community calls, peer-to-peer support chats, and one-to-one RTs/RMs calls. The curriculum focuses on the critical evaluation of emerging scientific literature, strategies to write constructive reviews, understanding how systems of oppression manifest within the scientific enterprise, and helping mentees recognize and challenge their own internal biases that may influence their ability to remain objective when reviewing (Figure 3). In addition, we provide opportunities for peer-to-peer support, and mentorship and leadership training for those who wish to be mentors in future cohorts, an effort that will help make the program grow in its impact and reach.
During the 8-month self-guided component of the program (Figure 4), ARTs have access to training materials developed for the cohort-based component in the form of recorded videos, documents, and other resources that they can consume at their own pace. ARTs are paired with ARs and meet for 30-60 minute calls once a month. With the support and guidance of their mentors ARs, ARTs complete and publish 4 preprint reviews on PREreview.org. Prior to their first one-to-one call, ARs watch our recorded Mentor Thought Exercise video to encourage the development of a supportive, nurturing, and safe space for ARTs to develop their peer reviewing skills.
A note about the pilot
The PREreview Open Reviewers pilot launched in October 2020 with the cohort-based component of the program. Applications to join stage 1, 2, 3 were open for 2 weeks in September 2020 and shared with a limited number of people in our network. We received a total of 51 ECRs applications! For our pilot, we selected 10 applicants to enter stage 1, 8 applicants to enter stage 2 and 3, and identified 9 journal editors to enter stage 4. We hope to expand the number of ECRs in future cohorts as a number of non-selected applicants expressed a high desire to participate, demonstrating the need for such a program within the scientific community. Half of the selected applicants self-identified as underrepresented minorities (see below for what we mean by that). Stage 4 RMs are experienced journal editors who support our mission and are volunteering their time to train stage 1 RTs. Read more about our pilot participants here.
Our long-term goal with this program is to help address underrepresentation in peer review. Due to time zone and staff constraints, our pilot program mainly accommodates U.S. and European applicants and uses the National Science Foundation’s definition of underrepresented racial minorities in science, which includes Black/African Americans, LatinX, Indigenous/First Peoples/American Indians & Alaskan Natives. Thus, the designation of underrepresented minorities is defined using U.S. statistics rather than global statistics.
While we recognized this as a limitation towards our mission of empowering all voices who are underrepresented in scholarship, we believed that to be effective we needed to be specific about our inclusion goal for this first version of the program. Our goal for the future of this program is to scale our impact to include more research communities by openly sharing all our resources and know-how, and offering our support to organizations world-wide to adapt the materials and format to the needs of their communities empowering them to create their own version of the program.
As researchers who have trained in North America and have degrees in the life sciences, we also recognize our inherent cultural and scientific bias in how we developed this program. As our program continues to grow, especially as we expand it globally through partnerships with like-minded organizations that share our values, we will adapt our curriculum and training to address these biases. We understand and are committed to the necessity of continual reassessment in order to avoid recapitulating systems of oppression within our own programming.
To learn more about this program, give us constructive feedback, share ideas on how to improve it moving forward, please don't hesitate to contact us at email@example.com.
We thank the Mozilla Foundation for sponsoring the PREreview Open Reviewers pilot. We also thank the Mozilla Open Leaders community, in particular Abigail Cabunoc-Mayes and Chad Sansing for pioneering a completely online and interactive program that empowered people to be leaders in their communities from all over the world, and that provided us with the skills and inspiration to build our own peer review mentoring program.
We thank Drs. Emmy Tsang, Yo Yehudi, and Julia Lawndes for advising us on how to frame and organize a spin-off of the Mozilla Open Leaders, adapting it to our needs and those of the communities we want to serve.
We thank Dr. Andre Walcott for designing and co-facilitating the Mentor Thought Exercise.
We thank Maria Guerreiro, Journal Development Editor at eLife, for helping us connect with the eLife Editors who serve as stage 4 RMs in our pilot.
We thank Dr. Veronique Kiermer, Chief Scientific Officer at PLOS and member of the PREreview Steering Committee, for her advice and help connecting our team with PLOS Editors.
We thank Lindsay Morton, Senior Manager Contributor Outreach at PLOS, and Dan Morgan, Director of Communications & Community Relations at PLOS, for being our first guest speakers and sharing their knowledge with our RTs on how to be effective and constructive reviewers.