Last fall we launched Open Reviewers, a training and mentoring program that empowers early-career researchers (ECRs) to contribute to scholarly peer review. The program was designed to include four stages, enabling ECRs to enter at the stage that feels right for them and mature as advanced reviewers. The pilot started in the fall of 2020 with a cohort-based 14-week-long component (Stage 1)—which recently ended—and an 8-month-long self-guided component (Stage 2 and 3). In this blog post, we reflect on what we learned during Stage 1 of the program, and share ideas on what comes next.
Central to PREreview’s mission is to offer traditionally excluded researchers opportunities to build the skills, confidence, and network needed to engage in scholarly peer review. In creating Open Reviewers, we set the goal to build a training program that not only taught how to provide feedback on a research manuscript constructively, but also weaved into the very fabric of the curriculum tools to help identify and mitigate biases, reflecting on how systems of oppression (including but not liming to white supremacy, colonialism, and sexism) manifest in the review process.
Over the course of 14 weeks, ten ECRs (the mentees) met weekly and were guided through an equity-centered approach to reading manuscripts and writing constructive peer reviews. In addition to cohort-based learning, the mentees also received several 1:1 mentoring sessions, each paired with a journal editor (the mentors) who supported them through the writing of two reviews of preprints covering topics within the mentees’ field of work. For many, this was their first time writing a review.
Throughout the program mentees engaged in conversations that directly challenged the predominant way expertise is recognized in academia and peer review, where privilege and personal connections too often outweigh reviewing skills and experience in the reviewer selection process. As mentees said in their applications:
If we really want to diversify science we need to diversify the people who are labeled as "expert" in a field and [who] have the power to evaluate the quality of their peers' work. - Natalia Torres, PREreview Open Reviewer 2020
The importance of diversifying the pool of reviewers also means breaking down the Ivory Tower of science and the review process that happens "behind closed doors" to create a more open and equitable environment for reviewing and publishing scientifically rigorous papers. Radically changing the representation in the peer review process is an important part of creating an inclusive scientific community. - Gina Calco, PREreview Open Reviewer 2020
When it came to reviewing the first preprint, we asked mentees to read through a Bias Reflection Guide, a tool we developed to help reflect on and identify how our own assumptions and biases may lead us, the reviewers, to make a judgment on a research manuscript that is polluted with preconceptions. The guide is not meant to assess where each of us is in the process of recognizing and fighting bias, but rather provide a non-judgmental framework that can be used iteratively and to prompt open discussions.
Bias shows up everywhere and in subtle ways that are not always easy to recognize. One of our mentees shared with us that after reading the Bias Reflection Guide they realized that when they were choosing between two preprints to review, one authored by a research group at a US-based institution familiar to them and one authored by a group based in China, they instinctively chose the former. This experience led us to include this bias in the guide to reflect on how our thoughts can drive behaviors that contribute to the inequities present at the systemic level—e.g., privileging one preprint over another with valuable feedback that may enhance that groups’ research.
The mentors were also guided through similar discussions during what we called a Mentor Thought Exercise, in which we covered issues of power disparities and elements of non-violent communication centered around supporting their mentees. While organising this initial call, we were mindful of the busy schedule of the mentors and kept the discussion to one and a half hours. We later learned that many of the mentors preferred to have both more time dedicated to similar conversations and more engagement with the cohort of mentees in addition to their 1:1 encounters. This is something that we will make sure to incorporate into the next iteration of the program.
Towards the end of the program, we invited a group of expert guests to participate in two cohort calls, one focused on issues of equity, diversity, and inclusion in peer review, and one on editorial careers. These discussions inspired us and a group of mentees to dive deeper into some of these important topics and share our reflections with the broader community. Watch out for a series of blog posts coming out soon.
The program is designed to offer the mentees the opportunity to continue their learning experience in peer review by entering the next stage in the program, which allows them to write and publish four more reviews with the support of a mentor, their peers, and resources we make available. This program is also designed to empower the mentees to become mentors themselves in future cohorts—a framework we hope will enable this model to scale in its impact.
Below are some of their end-of-the-program reflections—words that encourage us to continue with this effort and engage more researchers from all over the world.
So, what’s next? The self-guided portion of this program is still ongoing with four mentees and four mentors paired up to write more preprint reviews and learn together. Our goal for the next few months is to reflect on the learnings from this pilot and incorporate them into our curriculum and program structure. We take feedback very seriously and are committed to the continual reassessment of the program through an equity lens in order to avoid recapitulating systemic oppression within our own programming.
We will also work with eLife, our newly announced partner, to share the materials we developed for this program with the broader community, including guides and videos that will be published over the course of the next few months for everyone to adapt, use and re-use!
We want to express our deep appreciation to those who participated in the program. We may have designed the program, but the participants provided value and richness. The interests of the mentors, mentees, and invited speakers go far beyond simply writing manuscript reviews; the participants inspired us through their commitment to creating more just and equitable spaces. We are thrilled to continue building inclusive and equitable communities with the current and future participants. We also thank eLife and Mozilla for sponsoring this pilot. We could have not done this without you!
