Innovation and technology in peer review: The case of Living with Machines

This year’s Peer Review Week theme, ‘Innovation and Technology in Peer Review’, has prompted us to reflect on open peer review – an umbrella term for a range of peer review models that build openness and transparency into the process to try to address some of the well-recognised issues with traditional anonymised peer review. Open review can involve revealing the identities of reviewers, sharing reviews publicly and/or encouraging broader community participation in the evaluation process. 

By running an open peer review process for one of our own open access book projects, Living with Machines, we hope to better understand the potential benefits of using digital tools and innovative approaches to peer review for the book manuscript’s development and evaluation, and to learn from the challenges that arise. 

The project: Living with Machines

Based on the hugely important research project of the same name, Living with Machines is at the cutting edge of digital history, exploring massive digitised historical collections to show the extraordinary impact of technology on humanity in Britain’s industrial revolution. Collectively written by the project’s large, multidisciplinary research team, the highly collaborative nature of the book makes it an ideal candidate for the kind of open peer review model we wanted to trial. 

The open review is taking place iteratively during 2024 and we are releasing early drafts of the book chapters for review in three phases. We have invited a cross-disciplinary review panel comprising ten reviewers from nineteenth-century studies, history and digital humanities to participate, each of whom has agreed to waive their anonymity in the feedback they provide on the draft materials.  

The role of technology in the review process

Technological innovation is central to the successful running of this open peer review model. The process relies on our instance of the scholarly publishing platform, Manifold, to facilitate the transparent sharing of drafts and commenting functionality. Using Manifold’s reading group feature, we have invited reviewers to leave their comments as annotations directly on the draft chapters in the platform which are visible to, and can be commented on by, the other reviewers and the author team. By using Manifold, this experiment has turned what is typically a private dialogue into a more collaborative, open discussion. The platform’s commenting tools also helps to target the review feedback and make it easier to refer back to activity in the reading group when revising materials.  

One immediate benefit of using the open access Manifold platform to conduct the review is that we have been able to share the draft chapters publicly at the same time (although the reviewers’ comments are currently hidden from public view). Giving other researchers access to draft materials and findings at this early stage before publication can help improve the speed and efficacy of research dissemination.  

What have we learned so far?

We have carried out the first of the three review phases so far and, while it is still early in the process, we have already learned a lot: 

• The opportunity for dialogue between the reviewers, and between authors and reviewers, is one of the most beneficial aspects of the approach as certain points can be amplified or alternative views presented in a conversational format. 

• Although experimental in nature, the book project still needs to fit into our traditional workflows, so careful planning and a closer working relationship than normal between our editorial and production team members are key to ensuring its success. 

• The process is relatively new to us, the author team and reviewers, so it is important to make sure we are all on the same page in terms of project expectations. We developed guidance, ran workshops and went over the process in detail beforehand, including technical aspects of the platform. Spreading the workload out into phases also meant we have been on hand to answer any queries that might come up during the review process. 

• Allowing a lot of run up to the project has proved important, setting up the reviewer panel in advance to facilitate the review once the material is submitted and published in draft form.  

• In terms of the review process itself, we had to be open to how authors and reviewers found it easiest to interact with the platform itself, updating our guidance as necessary. The feedback collation and revision process is also tested by the model, and we will be continuing to learn from authors and reviewers to pass on feedback to the Manifold team and update our own processes.  

• As mentioned, the review is being carried out by a small invited reviewer panel, but we would also be keen to explore wider, public open review for a future project. We know authors may be apprehensive about this, and the pressure to respond to a larger, more public audience could also lead to delays in the revision process, but we hope to learn from our current experience and incorporate our learnings into future experiments in open access. 

• Lastly, long-term preservation is an outstanding question for us. Draft materials with comments do not easily fall into the categories of preservation that we have for our other open access monographs at present. 

Ultimately the greater inclusion of interdisciplinary and diverse perspectives means that the Living with Machines book project will be scrutinised from multiple angles, hopefully allowing the authors to benefit from this breadth of early engagement with their work.

We are looking forward to taking our learnings into the next development and review phases for Living with Machines.  

We’re continuing to reflect more broadly on the benefits and challenges of both open and traditional forms of peer review in the evolving scholarly publishing landscape, and in the context of our press’s ethos to democratise access to knowledge. By leveraging digital tools to foster transparency, collaboration and inclusive approaches in peer review, this project highlights the scope for innovation in this area and we’re excited to build on this model for future works.  

Has this post made you feel any differently about open peer review? If you’d like to talk to a member of our team about this process, or conducting open peer review on a new project, please get in touch.