Accepted on 25 Oct 2021
Submitted on 08 Aug 2020
‘…the major discoveries made by Albert Einstein or Francis Crick and James Watson were not subjected to 21st-century standards of peer review. History is the ultimate arbiter in science, and has no need for pre-publication peer review.’ (Fisher & Parisis, 2015)
Post-publication peer review (PPPR), in contrast with the traditional peer review model, is when peer review takes place, in part or in whole, after a journal article has already been published (Ford, 2013). PPPR can broadly be divided into two categories: Primary and Secondary (Markie, 2015). Primary PPPR is when the whole of the peer review activity takes place following the publication of the article, thus replacing the pre-publication peer review process. Secondary PPPR, on the other hand, is complimentary to the traditional peer review process, as review takes place both pre- and post-publication.
For this review, MEDLINE, EMBASE and Web of Science databases were searched using the terms ‘post-publication’ AND ‘peer review’ within both Titles and Abstracts on 26-Mar-2020 with no filters for language or date restrictions. The database searches yielded the following number of articles – MEDLINE: 717; EMBASE: 85 and Web of Science: 152. A similar search was undertaken of journals related to the field of scholarly communications identified here: https://osc.cam.ac.uk/about-scholarly-communication/publishing-options-research-scholarly-communication This search yielded an additional 33 articles. However, very few articles provided insights into the current status, benefits and limitations of PPPR, thus providing the rationale for this paper. Results related to the various forms which facilitate PPPR were synthesised and are summarised in this paper. Information regarding the mechanics of a specific PPPR platform was gained by exploring the information provided on their respective website. PPPR represents a significant evolution in the peer review process, and one which, if extensively implemented, has the potential to revolutionise academic publishing and scholarly communication as we know it. However, few publications have sought to explore the various forms of PPPR and evaluate their potential impact.
This review paper will examine various forms of PPPR, as it relates to the journal publication system, and their respective merits and limitations. This discussion will focus mainly on the medical and scientific publishing domains. This review explores what the benefits of promoting PPPR are and the recipients of these benefits. The review also reflects on the potential for PPPR to change the scholarly communications landscape in terms of fostering more collaboration and timely exchange of information, and therefore naturally complementing the open research environment. Finally, the review also touches on what changes would be needed in the reward system for peer reviewers in order for PPPR to function effectively.
Scholarly publishing is undergoing rapid changes as preprint production becomes more commonplace and open access dissemination becomes mandatory and/or desirable by funding agencies. With preprint publishing, researchers can now share their research findings before they have been examined and scrutinised. Pre-print servers represent a rapidly growing component of the open science movement and involve the addition of a manuscript to a publicly accessible online server without the usual process of peer review. The main advantage of using pre-print servers is the speed with which articles can be made publicly available, enabling the research community to immediately benefit from and build upon their findings. Another important aspect of preprint servers is their accessibility. One of the ongoing criticisms of academic publishing is the difficulties accessing articles from behind the paywall. Pre-prints servers have been heavily utilised during the COVID-19 pandemic, with estimates of 39.5 COVID-19 pre-print manuscripts being deposited per day (Brierley, 2021). Fraser et al’s study also indicated that COVID-19 pre-prints are utilised 15 times more than non-COVID-19 pre-prints (Fraser et al., 2020).
The Open Science movement aims to ‘extend the principles of openness to the whole research cycle’ (de la Fuente, n.d.) and includes, among other aspects open data, open access, open source and open methodology (Aleksic et al., 2014). Open Science is touted as a mechanism to improve transparency in research design and reporting, to enhance accessibility of research findings and to foster greater collaboration between researchers (Moher et al., 2016). Furthermore, research articles can be read by a wider audience—including researchers who do not have subscriptions to paywalled journals, and also the public, including journalists, in general.
Open peer review, where the peer review reports (‘open reports’ or ‘signed peer review’) and sometimes the identity of the reviewers (‘open identities’) are published openly, is another component of the Open Science movement and aims to promote transparency and accountability in the peer review process (T. Ross-Hellauer, 2017). Several authors have reported that open peer review maintains (van Rooyen et al., 2010; van Rooyen et al., 1998; Vinther et al., 2012) or indeed improves the quality of review (Bornmann et al., 2012; Walsh et al., 2000). Open peer review has been trialled and adopted by some publishers, predominantly within health sciences (Wolfram et al., 2020). Another advantage of publishing the reviewers’ names and reviews alongside the original paper is that the reviewers are rewarded for their time and expertise. This is an important consideration, given the time and resources which go into peer review, primarily by academic staff who already have significant teaching and research commitments. Some researchers feel that peer review activity, which, while critically important, can be onerous (Kelly et al., 2014) and should be considered when applications for promotion are considered and open review could help to facilitate this (Nicholson & Alperin, 2016).
Despite advancements, the process of journal peer review has largely remained the same. One must ask, however, whether the traditional pre-publication peer review process is still fit for purpose in the changing publishing environment? And what are the alternative approaches of journal peer review?
Journal peer review has a number of main objectives. One is to ascertain whether a piece of work meets the required standard of quality and originality for publication in a specific journal (Mayden, 2012), in line with its remit and editorial policies. In other words, the peer review process seeks to provide a quality mark which can be an indicator of reliability to non-expert readers (Jubb, 2016). Another objective of peer review is to ascertain if the paper would appeal to the journal’s audience (Kelly et al., 2014). Finally, peer review has an important curatorial function whereby the authors implement feedback from editors and peer reviewers, sometimes as part of an extensive and often iterative process, in order to refine and improve their manuscript.
The traditional peer review model has two characteristics which are pertinent to this discussion. Firstly, unless the reviewer voluntarily signs the review, the authors are usually not aware of the identity of their reviewers (single-blind) and often vice versa also (double-blind), with the aim of reducing bias (Okike et al., 2016). Secondly, peer review takes place prior to the publication of the article, meaning the peer review process is the gateway to dissemination of information to the research community and plays a role in ensuring the quality and validity of published research. This also means that reviewers and editors can be regarded as the ‘gatekeepers’ of the peer review process.
There are recurring issues with the traditional pre-publication peer review system, such as conservatism and biases (see, for example, (Lee et al., 2013; Luukkonen, 2012; Johnson & Dickersin, 2007) not to mention questions about reliability and validity (see, for example, Bornmann, 2011). Another significant criticism of the traditional pre-publication peer review system is that it is often lengthy, resulting in delays in the dissemination of new findings which would benefit the research community and ultimately advance the progress of science and medicine (Peters, 2014). An ideal peer review system would be fair and free of bias, providing timely feedback to authors in a constructive manner, while upholding standards of research integrity and scientific methodology. Macmillan Palgrave surveyed 403 Nature Publishing Group and Palgrave Macmillan authors, 67% of which agreed or strongly agreed that ‘publishers should experiment with alternative peer review’ (Bourke-Waite, 2015), indicating that there is an appetite for change in the current peer review system. Notwithstanding the innovations of new procedures and technologies, the adoption of new mechanisms and platforms of peer review have been sluggish (Horbach & Halffman, 2019). For example, PubMed Commons was an online tool established by the National Institutes of Health (NIH) to comment on articles published in in PubMed. This platform was discontinued, citing ‘low levels of engagement’ (Dolgin, 2018).
With the changing scholarly publishing landscape, there are opportunities to reform and redesign peer review models and infrastructure. The use of preprints, especially during the COVID-19 pandemic, has shown that there are many benefits for scientific results to be published before traditional peer review. The age of open science also calls for new research infrastructure that can incorporate vibrant and rigorous discussions that allow for scientific progress, whilst striving for equality in scholarly communication. In addition to open peer review, post-publication peer review (PPPR) provides a model that is most beneficial for knowledge domains where the immediate dissemination and exchange of scientific results is crucial. In the rest of the paper, three major forms of PPPR are examined: letters to the editors/commentaries, journals which support primary PPPR, and academic social networks, followed by a discussion of the benefits and constraints of PPPR.
The literature revealed that PPPR can take many forms. In order to narrow the scope of this discussion, we have focused upon more formal forms of PPPR and exclude informal reviews and discussions on mainstream social media such as Twitter and LinkedIn. The main kinds of PPPR identified in the literature were journals which undertake primary PPPR, Letters to the Editor or Commentaries and academic social networks which permit PPPR.
While most publishers and journals continue to employ the traditional model of pre-publication peer review, a small number have recently ventured into primary PPPR, where all of the peer review activities take place after publication. A key advantage of journals which use the primary PPPR model is the speed with which articles are disseminated, as the often-lengthy pre-publication peer review step associated with traditional journals is omitted. Reduction in the time between submission and publication facilitates the swift dissemination of the latest findings into the research community (Huisman & Smits, 2017). This may be of crucial significance in rapidly emerging situations, where there is an urgent requirement for clinical, scientific or epidemiological data such as we have seen in the current COVID-19 pandemic (Rubin et al., 2020). Of course, reduction in time must be balanced against the need for careful evaluation of study design, methodology and reporting.
F1000 Research online platform is an example of a journal which facilitates both primary and secondary PPPR (http://f1000research.com/; Liao & Xie, 2012). This open-access platform invites reviewers on the author’s behalf after online publication of the article, and both the peer review comments and the author’s responses are published. All reviewer’s names are also published, along with their relevant area(s) of expertise. At each stage of the PPPR process, the article is clearly delineated as ‘awaiting peer review’, ‘approved with reservations’, approved by one or more reviewers, rejected etc. Following full approval, other readers can post comments, which are not pre-filtered by the platform, although they do reserve the right to remove any offensive content. Researchers can also set up alerts to track the PPPR process of particular articles. Many authors use this platform to publish articles in non-traditional formats, e.g., Open Letters, or protocols for studies, trials or systematic reviews. The article processing charge (APC) applied to the publication is tiered, depending on the volume of editorial work involved. The Irish Health Research Board (HRB) use the F1000 PPPR model to publish articles on their HRBOpen platform, where a key advantage is open access publication with no APC for any researchers who have held a HRB grant or worked on a HRB funded/co-funded project on or since 2017 (HRB Open).
MedEdPublish is another platform which uses the primary PPPR model, specifically within the field of medical and health professions’ education (MedEdPub). Once an article has been checked to ensure that it meets the minimum standard for publication, it is assigned a Digital Object Identifier (DOI) and published online. The minimum standards include adhering to a sufficient standard of written English, being in an acceptable format in accordance with the style of the journal and including an ethics statement, where applicable. Reviewers, which may include a MedEdPublish editor then openly publish their reviews. Articles are demarcated as ‘Not yet Peer Reviewed’, ‘Author revision recommended’, ‘Peer reviewed and of acceptable standard for indexing’.
The journal PLOS One employs a variation of this process in that editors and reviewers determine if an article should be published on the basis of methodological soundness, including ensuring the publication reports previously unpublished work carried out to a high technical standard, written in understandable English and citing adherence to appropriate ethical standards. Then following publication, readers can make a comment online (Hames, 2008). Science Paper Online, an initiative from the Chinese Ministry of Education appears to be a hybrid of pre- and PPPR, where papers are checked by the editor for quality, methodology and presentation before being published online immediately (Ren, 2013). However, in this case, authors can opt in or out of peer review.
Letters to the Editor or Commentaries are one of the longest-standing forms of secondary PPPR (peer review which takes place in addition to pre-publication peer review). Some journals also permit online comments to be contributed, although it seems that this function is not frequently used in the academic literature. The International Committee of Medical Journal Editors (ICMJE) advocates that ‘Medical journals should provide readers with a mechanism for submitting comments, questions, or criticisms about published articles, usually but not necessarily always through a correspondence section or online forum.’ (Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals). One of the advantages of Letters to the Editor is that most journals assign them a DOI so that they can be referenced and cited. However, as Shashok and colleagues found, the journal where the original paper was published may decline to publish the Letter or Commentary. Some authors have raised the issue of where PPPR should be published and ask whether journals should be obliged to publish Letters to the Editor relating to papers that they have published (Shashok & Matarese, 2018).
There is also the issue of accessibility: if a Letter to the Editor is published in a journal which is behind the paywall, some researchers will not be able to accurately examine the topic under discussion (Winker, 2015). Readers may even have to pay twice – once for the original article, and a second time to read the Letter to the Editor relating to the original article (Winker, 2015), illustrating that PPPR is not an issue which can be tackled in isolation. Shashok and colleagues report on their many attempts to publish a response to a consensus statement originally published in BMC Medicine (Shashok & Matarese, 2018). After their correspondence piece was declined by BMC Medicine, they approached a second journal, but despite a request for a waiver of the article processing charges, they were only granted a 50% reduction, meaning this option was not affordable. Eventually their response was published with relative ease through F1000Research. This situation begs the question: how many relevant readers will have access to the PPPR if it takes the form of Letters to the Editors or Commentaries?
Others have also pointed out that letters are limited in length and number of references which can be used (Altman, 2005; Anstey, 2014; Biagioli & Lippman, 2020). This makes it difficult for a PPPR to sufficiently argue a point in this format. Altman points out that there may also be a limitation in terms of the time frame allowed for PPPR within journals (Altman, 2005), with reviewers being required to submit PPPRs within a pre-determined time period in order for them to be considered for publication. Also, if the peer review process for Letters to the Editor is protracted, as peer review frequently is, will the letter have the desired impact? Finally, letters to the editor are not indexed by Clarivate Analytics, who decide the journal impact factors, so researchers are not incentivised to actively participate in PPPR.
One of the advantages of both primary and secondary PPPR is that it can facilitate an ongoing, dynamic discussion between researchers and online academic social networks provide a platform for this discussion. Academic social networks are online forums where researchers can connect and include the tools typically available to social networks including having a public or private profile (in this case an academic or professional profile), messaging and forming groups (Serantes, 2017). Academic social networks provide the benefit of review by multiple peers, not just a selected two or three. It is hoped that a broader inclusion of participants in the peer review process would facilitate a more robust, and well-rounded discussion of the topic. This may be particularly relevant given that many fields of research are increasingly becoming interdisciplinary, so different reviewers could comment on specific areas that relate to their expertise e.g., statistics, methodology, ethics or a clinical specialty. In addition, if the PPPR is published openly by named users on academic social networks, it may provoke more meaningful and balanced engagement with the topic. Given the trend toward online academic conferences due to environmental concerns, or more recently because of the COVID-19 pandemic, it is perhaps fitting that discussions between researchers should be facilitated via online PPPR, rather than in person. One of the benefits of using academic social networks, compared to traditional social media is not being limited to e.g., a certain number of characters – this can facilitate a more in-depth discussion (Lewis & Rallon, 2013). Another benefit of PPPR taking place on academic social networks is the easier engagement with early career researchers, rather than relying on email addresses which change frequently due to the transient nature of research employment.
PubPeer is one commonly used example of an academic social network and was developed in 2012 with the aim of facilitating discussion of scientific articles after their publication, thereby allowing researchers to engage in ongoing scholarly discourse. PubPeer is promoted as an extended Journal Club, as authors are sent an email notifying them of comments made on their publications and it allows contributors to comment anonymously or openly on published articles or pre-prints (Townsend, 2013) and is not specific to any one journal. Users can create an account, either anonymously or using their name and by searching for an article using a PubMed Reference Number (PMID), Digital Object Identifier DOI or arXiV Identifier (arXiVID), can leave a comment on the article. When registering, the user is asked to input the details of a relevant paper they have authored, however if they have not authored a suitable paper, they can contact PubPeer to set-up an account. PubPeer has however, been criticised because it seems that contributors use the anonymous function to make comments in an unconstructive manner, including posting overly negative or even malicious comments (Biagioli & Lippman, 2020; Blatt, 2015). Blatt further argues that the focus appears to be on minute presentation details, rather than on methodology (Blatt, 2015). However, Bordignon compared the rate of retractions generated by comments on PubPeer compared with those brought about by negative citations and concluded that PubPeer is more influential to the ‘correction of science’ than negative citations (Bordignon, 2020). He attributed the higher engagement on PubPeer, compared to other platforms, to the fact that comments on PubPeer can be made anonymously. In 2014, researchers from the Chinese University of Hong Kong and the University of California published a review of a Nature publication on PubPeer, pointing out a possible methodological error (Tony Ross-Hellauer, 2017). The review, which accumulated 40,000 views on PubPeer is credited to have initiated investigations of the lead author on the paper, who was later found guilty of scientific misconduct (Science self-corrects – instantly). The article was later retracted (Obokata et al., 2014) clearly illustrating how academic social networks can be used to provide swift oversight of academic publications via PPPR.
Open Review, using the Research Gate platform provides a similar forum which facilitates more formal, structured reviews of previously published articles with the aim of determining if the research is reproducible (Megwalu, 2015). Since reviews are tied to a user’s Research Gate account, readers can view the reviewer’s credentials and publications. Open Review also allows authors to request feedback on their publications. A reader also used this platform to successfully question the reproducibility of results in the Nature paper mentioned above, also contributing to the exposure of scientific misconduct (Cossins, 2014).
ScienceOpen is another academic social network forum (Review on ScienceOpen, 2020), where users can comment on millions of academic articles and can search for articles and filter them for popularity (such as Altmetrics, citations etc). However, in this case, reviewers cannot post anonymously and are required to provide a link to their Open Research and Contributor Identification (ORCID). In order to publish a comment, readers must have at least one publication associated with their ORCID account, while in order to post a review or rate a publication, Scientific Members and Expert Members must have a minimum of five publications (What can a Researcher do on ScienceOpen?, 2020). ScienceOpen reviews are now assigned a DOI on publication, which means that they can be located, and cited (Winker, 2015) and because the reviewers are identified it provides a record of peer review activity, which may give an incentive for reviewers. However Allen and Dawson (2015) point out that PPPR at ScienceOpen needs to be included in Altmetrics in order to gain momentum (Allen & Dawson, 2015).
Another similar platform, TrueReview, aims to facilitate the swift dissemination of articles to research communities but also to incentivise peer review by publicly acknowledging the efforts of peer reviewers (de Alfaro & Faella, 2016). Users who provide timely and thorough peer review have the option of linking their TrueReview to a website of their choice, e.g., their affiliated institution. For example, the first review of a paper is considered valuable, as is providing a rating which contrasts with previous ratings but which is later established as correct.
In summary, PPPR can take place via a range of mechanisms, including journals which carry out primary PPPR only, Letters to the Editor/Commentary, and via academic social networks. These forums provide users with opportunities to effectively engage in PPPR, either openly or anonymously. However, how do researchers decide which one to use and does the variety of options improve the accessibility of PPPR or diminishes it (Van Noorden, 2014)?
Poole contends that publication should mark the beginning, not the end, of the peer review process (Poole, 1996). Advocates of PPPR also point out that pre-publication peer review cannot identify all methodological weaknesses or instances of scientific misconduct (Bordignon, 2020). PPPR provides ongoing opportunities for issues to be addressed. However, there are a number of factors related to PPPR which still need to be examined.
Firstly, is it always beneficial to disseminate research findings so quickly into the research community? It is widely acknowledged that one of the key advantages of primary PPPR is avoiding the delays which are typically synonymous with the pre-publication peer review process. However, improving the speed of the peer review process may not necessarily be a good thing, for instance, Baverstock notes that it may result in a reduction in quality (Baverstock, 2016). Publication of research findings before experts in the field have had a chance to review them in detail may also result in misinterpretation as results are often nuanced and may require expert knowledge to accurately appraise, exposit or contextualise them. This is particularly true where the mainstream media disseminate findings which have not yet been interrogated closely, which we have seen during the COVID-19 pandemic. However, this caution must be carefully balanced against the urgent need to publish, for example, epidemiological, medical or scientific data which may be of critical importance in an emerging situation, such as a pandemic, or in order to promote swift advances in knowledge of diseases and their treatment.
There is also ongoing debate as to whether PPPR should be anonymous (Teixeira da Silva & Blatt, 2016). As mentioned previously, with traditional pre-publication peer review, the author is usually blinded to the reviewer’s name and sometimes vice versa. In the case of Letters to the Editors/Commentaries, it is customary for PPPR to be published openly, identifying the author(s). However, in the case of journals which use the primary PPPR model or with academic social networks, either open or blinded review is possible. The main advantage of open reports, where the peer review report is published alongside the article and signed reports, or indicating the peer reviewers’ names, is increased transparency. Signed reports may also encourage reviewers to provide more constructive feedback. However, signed reports may also act as a deterrent to honesty and early career researchers may be reluctant to contribute negative feedback for fear of reprisal. On the other hand, if prior registration to an academic social media platform is required before a user can provide PPPR, this may act as a deterrent for trolls and thus maintain the standards of PPPR (“Post-Publication Peer Review of Scientific Manuscripts: Boom or Bust?,” 2018). Alternatively, commenters could be required to link to their ORCID account, so that their research credentials could be easily viewed.
There is also the question of providing incentives for academics to engage in PPPR. Whilst the pre-publication peer review model is well established; it is under increasing pressure as the volume of the scientific literature increases and the availability of peer reviewers dwindles. Traditionally, publications, and not reviewing, is the largest criteria for promotion, so (unpaid!) peer reviewing may be regarded as a luxury which many academics feel they can ill afford. While it is recognised that without change in the hiring criteria for academics that recognises the often-significant contributions of reviewers, it is hard to see how this problem can be overcome. However, perhaps this is a step in the right direction. Open peer review, whether it takes place pre or post-publication can provide a record of peer reviewing activities which academics have undertaken, which is perhaps a step towards recognition of this valuable role (“Open Peer Review,”).
Streamlining or integrating the different mechanisms of PPPR could promote more structured and traceable engagement, but this aspiration has not yet been realised.
Regardless of what forum is used, a robust system is needed to facilitate PPPR, so that the original authors and other researchers have access to and can benefit from the perspectives of their peers. Bastian points out that PPPR can be ‘highly fragmented’ as there are multiple platforms and forums (Bastian, 2014) – in addition to the ones described in this review, there are comments published on conventional social media, blogs, pre-print servers etc. In the absence of a clear structure or platform for PPPR, providing peer review can be onerous and ultimately ineffective. It is unclear as to whether many researchers are aware of the various PPPR opportunities that are available to them? Should publishers be promoting them more?
While PPPR is gaining momentum, it doesn’t yet have the same ‘status’ as pre-publication peer review. Jubb’s interviews with publishers indicated that they were in favour of PPPR overall, as there is the potential for discussions which may have taken place between two people in the past to now include a larger group. They also saw the value of communication between researchers, editorial teams and reviewers and felt that it is important to capture and quantify PPPR and its impact. However, they felt it would supplement, and not replace the traditional peer review model (Jubb, 2016). It is also important to consider whose responsibility it is to implement and provide governance over the PPPR system? The key stakeholders are clearly researchers, editors and publishers but it is unclear as to whether any of these groups have the appetite or the necessary time to provide this oversight? In order for PPPR to add meaningful value to the field of scholarly communications and become a sustainable vehicle for academic collaboration, consideration should be given at the policy level as to how PPPR is implemented.
It seems apparent that peer review in its current form is unsustainable, but the replacement for, or the next iteration of, peer review is as of yet unclear. The advantages of PPPR include the opportunity to include a wider volume and variety of participants, and increased speed of publication and perhaps greater transparency. The potential of extensively implemented PPPR practices if realised, may facilitate robust, high-quality debate and enhance collaboration between researchers. However, the paywall and the lack of a structured and integrated platform for facilitating PPPR discourse, frequently stand in the way of an efficient and effective PPPR process. It also remains to be seen as to whether open or anonymous PPPR is more advantageous.
This work was supported by the Health Research Board Trials Methodology Research Network (HRB-TMRM) as part of the HRB-TMRN-2017-1 grant. The research was conducted independently from the funder, and researchers are independent from the funder.
The authors have no competing interests to declare.
Paper concept: LOS, PD and LM. First draft of paper: LOS. Funding acquisition: PD. All authors approved the final version of the paper before submission.
Aleksic, J., Alexa, A., Attwood, T. K., Chue Hong, N., Dahlö, M., Davey, R., Dinkel, H., Förstner, K. U., Grigorov, I., Hériché, J.-K., Lahti, L., MacLean, D., Markie, M. L., Molloy, J., Schneider, M. V., Scott, C., Smith-Unna, R., Vieira, B. M., as part of the AllBio: Open, S., & Reproducibility Best Practice, W. (2014). An open science peer review oath. F1000Research, 3, 271–271. DOI: https://doi.org/10.12688/f1000research.5686.2
Allen, L., & Dawson, S. (2015). Scholarly publishing for the network generation. Insights, 28(1). DOI: https://doi.org/10.1629/uksg.214
Altman, D. G. (2005). Unjustified restrictions on letters to the editor. PLoS medicine, 2(5). DOI: https://doi.org/10.1371/journal.pmed.0020126
Anstey, A. (2014). Letters to the editor: time for more scholarly debate. Br J Dermatol, 171(1), 1–2. DOI: https://doi.org/10.1111/bjd.13134
Bastian, H. (2014). A stronger post-publication culture is needed for better science. PLoS medicine, 11(12), e1001772–e1001772. DOI: https://doi.org/10.1371/journal.pmed.1001772
Baverstock, A. (2016). Is peer review still the content industry’s upper house? Learned Publishing, 29(1), 65–68. DOI: https://doi.org/10.1002/leap.1013
Biagioli, M., & Lippman, A. (2020). Gaming the metrics: Misconduct and manipulation in academic research. Mit Press. DOI: https://doi.org/10.7551/mitpress/11087.001.0001
Blatt, M. R. (2015). Vigilante Science. Plant Physiology, 169(2), 907. DOI: https://doi.org/10.1104/pp.15.01443
Bordignon, F. (2020). Self-correction of science: a comparative study of negative citations and post-publication peer review. Scientometrics, 124(2), 1225–1239. DOI: https://doi.org/10.1007/s11192-020-03536-z
Bornmann, L. (2011). Scientific peer review. Annual review of information science and technology, 45(1), 197–245. DOI: https://doi.org/10.1002/aris.2011.1440450112
Bornmann, L., Wolf, M., & Daniel, H.-D. (2012). Closed versus open reviewing of journal manuscripts: how far do comments differ in language use? Scientometrics, 91(3), 843–856. DOI: https://doi.org/10.1007/s11192-011-0569-5
Bourke-Waite, A. (2015). Innovations in scholarly peer review at Nature Publishing Group and Palgrave Macmillan. Insights, 28(2). DOI: https://doi.org/10.1629/uksg.243
Brierley, L. (2021). Lessons from the influx of preprints during the early COVID-19 pandemic. The Lancet Planetary Health, 5(3), e115–e117. DOI: https://doi.org/10.1016/S2542-5196(21)00011-5
Cossins, D. (2014). Setting the Record Straight. The Scientist: Exploring Life, Inspiring Innovation. Retrieved 29th January from http://www.the-scientist.com/?articles.view/articleNo/41056/title/Setting-the-Record-Straight
de Alfaro, L., & Faella, M. (2016). TrueReview: A Platform for Post-Publication Peer Review. arXiv preprint arXiv:1608.07878.
de la Fuente, G. B. What is open science? Introduction. https://www.fosteropenscience.eu/content/what-open-science-introduction
Dolgin, E. (2018). PubMed Commons closes its doors to comments: The US National Institutes of Health shutters its journal-commenting platform. Nature News. DOI: https://doi.org/10.1038/d41586-018-01591-4
Fisher, D., & Parisis, N. (2015). Social influence and peer review. EMBO reports, 16(12), 1588–1591. DOI: https://doi.org/10.15252/embr.201541256
Ford, E. (2013). Defining and Characterizing Open Peer Review: A Review of the Literature. Journal of Scholarly Publishing, 44(4), 311–326. DOI: https://doi.org/10.3138/jsp.44-4-001
Fraser, N., Brierley, L., Dey, G., Polka, J. K., Pálfy, M., & Coates, J. A. (2020). Preprinting a pandemic: the role of preprints in the COVID-19 pandemic. bioRxiv, 2020.2005.2022.111294. DOI: https://doi.org/10.1101/2020.05.22.111294
Hames, I. (2008). Peer review and manuscript management in scientific journals: guidelines for good practice. John Wiley & Sons. DOI: https://doi.org/10.1002/9780470750803
Horbach, S. P., & Halffman, W. (2019). Journal peer review and editorial evaluation: Cautious innovator or sleepy giant? Minerva, 1–23. DOI: https://doi.org/10.1007/s11024-019-09388-z
HRB Open. Health Research Board. Retrieved 30th January from https://hrbopenresearch.org
http://f1000research.com/. Retrieved 3rd May 2020.
Huisman, J., & Smits, J. (2017). Duration and quality of the peer review process: the author’s perspective. Scientometrics, 113(1), 633–650. DOI: https://doi.org/10.1007/s11192-017-2310-5
Johnson, R. T., & Dickersin, K. (2007). Publication bias against negative results from clinical trials: three of the seven deadly sins. Nature Clinical Practice Neurology, 3(11), 590–591. DOI: https://doi.org/10.1038/ncpneuro0618
Jubb, M. (2016). Peer review: The current landscape and future trends. Learned Publishing, 29(1), 13–21. DOI: https://doi.org/10.1002/leap.1008
Kelly, J., Sadeghieh, T., & Adeli, K. (2014). Peer review in scientific publications: benefits, critiques, & a survival guide. Ejifcc, 25(3), 227.
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2–17. DOI: https://doi.org/10.1002/asi.22784
Lewis, T. L., & Rallon, P. (2013). #BlueJC: BJOG and Katherine Twining Network collaborate to facilitate post-publication peer review and enhance research literacy via a Twitter journal club. Bjog, 120(13), 1699–1700. DOI: https://doi.org/10.1111/1471-0528.12451
Liao, X., & Xie, Y. M. (2012). F1000: a new medical literature evaluation and retrieval system. Zhongguo Zhong Xi Yi Jie He Za Zhi, 32(5), 701–703.
Luukkonen, T. (2012). Conservatism and risk-taking in peer review: Emerging ERC practices. Research Evaluation, 21(1), 48–60. DOI: https://doi.org/10.1093/reseval/rvs001
Markie, M. (2015). Post-publication peer review, in all its guises, is here to stay. Insights, 28(2). DOI: https://doi.org/10.1629/uksg.245
Mayden, K. D. (2012). Peer Review: Publication’s Gold Standard. Journal of the advanced practitioner in oncology, 3(2), 117–122. DOI: https://doi.org/10.6004/jadpro.2012.3.2.8
MedEdPub. Retrieved 30th January from https://www.mededpublish.org/What-is-post-publication-peer-review
Megwalu, A. (2015). ResearchGate: An academic social networking site. The Charleston Advisor, 17(1), 47–51. DOI: https://doi.org/10.5260/chara.17.1.47
Moher, D., Glasziou, P., Chalmers, I., Nasser, M., Bossuyt, P. M. M., Korevaar, D. A., Graham, I. D., Ravaud, P., & Boutron, I. (2016). Increasing value and reducing waste in biomedical research: who’s listening? The Lancet, 387(10027), 1573–1586. DOI: https://doi.org/10.1016/S0140-6736(15)00307-4
Nicholson, J., & Alperin, J. P. (2016). A brief survey on peer review in scholarly communication. The Winnower.
Obokata, H., Wakayama, T., Sasai, Y., Kojima, K., Vacanti, M. P., Niwa, H., Yamato, M., & Vacanti, C. A. (2014). Retraction Note: Stimulus-triggered fate conversion of somatic cells into pluripotency. Nature, 511(7507), 112–112. DOI: https://doi.org/10.1038/nature13598
Okike, K., Hug, K. T., Kocher, M. S., & Leopold, S. S. (2016). Single-blind vs double-blind peer review in the setting of author prestige. Jama, 316(12), 1315–1316. DOI: https://doi.org/10.1001/jama.2016.11014
Open Peer Review. (10th July). https://plos.org/resource/open-peer-review/
Peters, M. A. (2014). Open Science, Philosophy and Peer Review. Educational Philosophy and Theory, 46(3), 215–219. DOI: https://doi.org/10.1080/00131857.2013.781296
Poole, C. (1996). Invited commentary: evolution of epidemiologic evidence on magnetic fields and childhood cancers. Am J Epidemiol, 143(2), 129–132; discussion 133–126. DOI: https://doi.org/10.1093/oxfordjournals.aje.a008719
Post-Publication Peer Review of Scientific Manuscripts: Boom or Bust? (2018, 9th July). Enago Academy. https://www.enago.com/academy/post-publication-peer-review-of-scientific-manuscripts-boom-or-bust/
Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals. International Committee of Medical Journal Editors. Retrieved 10th January from www.icmje.org/index.html
Ren, X. (2013). Beyond online preprints: formalization of open initiatives in China. Learned publishing, 26(3), 197–205. DOI: https://doi.org/10.1087/20130308
Review on ScienceOpen. (2020). ScienceOpen. Retrieved 29th January from https://about.scienceopen.com/peer-review-guidelines/
Ross-Hellauer, T. (2017). Disambiguating post-publication peer review. Retrieved 30th January from https://blog.scienceopen.com/2016/09/disambiguating-post-publication-peer-review/
Ross-Hellauer, T. (2017). What is open peer review? A systematic review. F1000Res, 6, 588. DOI: https://doi.org/10.12688/f1000research.11369.2
Rubin, E. J., Baden, L. R., Morrissey, S., & Campion, E. W. (2020). Medical journals and the 2019-nCoV outbreak. N Engl J Med, 382(9), 866. DOI: https://doi.org/10.1056/NEJMe2001329
Science self-corrects – instantly. PubPeer Foundatation. Retrieved 30th January from https://blog.pubpeer.com/publications/36E5D01DFD3E874F721E607D0ADDD2#7
Serantes, A. (2017). Academic Social Networks: What They are and How They Can Help Science. Open Mind. https://www.bbvaopenmind.com/en/humanities/communications/academic-social-networks-what-they-are-and-how-they-can-help-science/
Shashok, K., & Matarese, V. (2018). Post-publication peer review in biomedical journals: overcoming obstacles and disincentives to knowledge sharing. RT. A Journal on Research Policy and Evaluation, 6(1). DOI: https://doi.org/10.31229/osf.io/8kxyz
Teixeira da Silva, J. A., & Blatt, M. R. (2016). Does the Anonymous Voice Have a Place in Scholarly Publishing? In Plant Physiol, 170, 1899–1902. DOI: https://doi.org/10.1104/pp.15.01939
Townsend, F. (2013). Post-publication Peer Review: PubPeer. Editors’ Bulletin, 9(3), 45–46. DOI: https://doi.org/10.1080/17521742.2013.865333
Van Noorden, R. (2014). The new dilemma of online peer review: too many places to post? Nature News Blog. Retrieved 30th January from http://blogs.nature.com/news/2014/03/the-new-dilemma-of-online-peer-review-too-many-places-to-post.html
van Rooyen, S., Delamothe, T., & Evans, S. J. W. (2010). Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial. BMJ (Clinical research ed.), 341, c5729–c5729. DOI: https://doi.org/10.1136/bmj.c5729
van Rooyen, S., Godlee, F., Evans, S., Smith, R., & Black, N. (1998). Effect of blinding and unmasking on the quality of peer review: a randomized trial. Jama, 280(3), 234–237. DOI: https://doi.org/10.1001/jama.280.3.234
Vinther, S., Haagen Nielsen, O., Rosenberg, J., Keiding, N., & Shroeder, T. (2012). Same review quality in open versus blinded peer review in” Ugeskrift for Læger. Dan Med J, 59(8), A4479.
Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000). Open peer review: a randomised controlled trial. Br J Psychiatry, 176, 47–51. DOI: https://doi.org/10.1192/bjp.176.1.47
What can a Researcher do on ScienceOpen? (2020). ScienceOpen.com. Retrieved 30th January from https://home.scienceopen.com/693-2/#more-693
Winker, M. (2015). The promise of post-publication peer review: how do we get there from here? Learned Publishing, 28, 143–145. DOI: https://doi.org/10.1087/20150209
Wolfram, D., Wang, P., Hembree, A., & Park, H. (2020). Open peer review: promoting transparency in open science. Scientometrics, 125(2), 1033–1051. DOI: https://doi.org/10.1007/s11192-020-03488-4