Research evaluation encompasses the practices of assessing research quality and impact of scholarly works ex ante and ex post. Ex ante research evaluation usually refers to the evaluation of research proposals for grant funding, where the quality, feasibility and potential contributions of funding proposals are assessed. Ex post research evaluation, on the other hand, is used to assess scientific-scholarly and sometimes economic and societal impacts, after a research project has been conducted. At the individual level, the assessments are often used in the decision-making process of hiring and promotion of scholars and of their career advancement, as well as the evaluation of grant proposals and awards. At the university level, research evaluation is sometimes used for allocating block grants to universities. While research evaluation is considered necessary to assess the research performance of individuals and universities, there have not been ethical guidelines in the drafting of evaluation processes or criteria notwithstanding the constitutive effects of evaluation (Dahler-Larsen, 2012). A survey of national evaluation systems (Ochsner, Kulczycki and Gedutis 2018) reveals that different systems have incompatible priorities for the research evaluation. As a result, their evaluation criteria vary, e.g. metric and non-metric require diverse approaches, including the ethical ones.

Whitley (2007) argues that universities and centers of research are often in competition for favorable assessment and that strong research evaluation systems can limit intellectual autonomy and the ability to implement research strategies that challenge current orthodoxies. Moreover, research evaluation may impact on the development of disciplines and limit novelty and inventiveness (Whitley, Gläser and Laudel 2018). In many scientific fields highly cohesive scientific elites may influence the organization of strong Research Evaluation Systems (RES) in agreement with their conceptions of quality. Where elites hold the consensus on central topics of their disciplines, strong RES may reinforce their authority, as they might decide the quality standard for the discipline. Research evaluation plays a fundamental role in both the development of disciplines and the career advancements of researchers. It is expected to impact on the development of scientific fields, as it may limit novelty and inventiveness of emerging researchers, which must conform to the dominant elites to achieve academic consensus.

There are also known problems and issues concerning research evaluation—both in the peer review process and the criteria used, including citation-based metrics. The assessment in Social Sciences and Humanities (SSH), for instance, may be based on emotions and on the interactions between individuals; the social identity of reviewers and their membership to a scientific-scholarly community, rather than the neutral judgment, can play a fundamental role (Lamont 2009). For one, bias in peer review has been discussed with respect to gender, race, language, career stage and interdisciplinarity (see, for example, Helmer, 2017; Lee, et al., 2013). Peer reviewers also tend to be conservative and risk-averse in their evaluation of innovation methods and approaches (Luukkonen, 2012), not to mention the inconsistent reliability and validity of peer review notwithstanding the availability of innovative procedures and platforms (Bornmann, 2011; Horbach & Halffman, 2019).

Furthermore, studies have shown that the use of citation-based metrics have led to the misuse and gaming of evaluative metrics (see, for example, Biagioli & Lippman 2020) as well as changes in research practices and knowledge production (de Rijcke et al., 2016). It is understood that the use of metrics induces competition, rather than collaboration, between researchers. The drive to publish in high JIF journal also prompts researchers and scholars to publish in international journals, leading to decreased number of publications in local/national languages that are important especially for the SSH. Some have argued that the use of metrics, which eventually has led to ‘misuses’ and ‘abuses’ of metrics, is due to the audit culture, in which accountability is at its core. Ma and Ladisch (2019) have suggested that evaluation complacency and evaluation inertia are a cause, as well as an effect, of the use of metrics in research evaluation. Recently, there are increasing pressures for institutions to reconsider and reconfigure the use of citation-based metrics in response to DORA (ASCB, 2013), The Metrics Tide (Wilsdon et al., 2015), The Leiden Manifesto (Hicks, et al., 2015) and the Hong Kong Principles (Moher, et al., 2020).

Taking into account the complexities of research evaluation, we must consider whether to look “from above” and seek universal ethics, or to calibrate our optics for an empirical case study. At the initial stage of our endeavour it is more viable to seek for a theoretical horizon than limit ourselves to empirical case studies. Merton (1973) has proposed scientific ethos often known by the name of CUDOS: communalism (originally communism), universalism, disinterestedness, and organized skepticism. While his conceptions are closely related to the goal of science and scientific method, there seems to be a lack of ethical justification. Therefore, CUDOS are open to criticism for being too general, not reflective enough and rather inefficient if compared with the particular practices of scientific research in their diversity. For this reason, a more detailed study of the ethical field is needed. In the rest of the paper, we will argue that ethics of research evaluation lies in the overlapping area of research ethics and evaluation ethics, followed by a discussion of three ethical theories: deontological ethics, consequentialist ethics, and virtue ethics. Finally, we propose that best practices of research evaluation can be based on a mixed approach.

In this section, we review major documents concerning research ethics and integrity, on the one hand, and evaluation ethics, on the other, to situate ethics of research evaluation in the overlapping area of these two domains (Figure 1).

To address ethical issues of research evaluation, it is necessary to look at the ethical theories and instruments they provide. Studying traditional ethical approaches, Doris and Stich (2007) attempt at broadening the methodological scope of philosophical ethics. They argue that traditional ethical theories can benefit from an empirical approach toward the solution of ethical issues. It is thus possible that new ethical approaches, as well as introduction of implicit and tacit ethical knowledge, can provide us with explanatory mechanisms needed in our pursuit for considering ethics of research evaluation. Looking back at the theoretical tradition in ethics, which is applicable to research in general and research evaluation in particular, the general ethical choices are limited to certain number of theoretical approaches. The ‘hard-core’ of normative ethics, is comprised, first of all, of deontological ethics and consequentialist ethics, and, to a lesser extent, virtue ethics.

One of the crucial issues of the research evaluation ethics is its borders and scope of the field. In other words, it is the question of what is ethical. Or, how to clearly distinguish between, let’s say, the epistemological and ethical? According to Mustajoki and Mustajoki (2017), recognition of ethical questions comprises of a three-fold way: (a) identification of stakeholders (e.g., individuals, groups, communities, animals, ecosystems, future generations etc.), (b) understanding rights and responsibilities of the stakeholders and for the stakeholders, and (c) definition of options, i.e. looking for the win-win situation, or at least proximity to it, for the stakeholders involved. Although Mustajoki and Mustajoki (2017) do not refer to the idea of common good, the three criteria of the ethical presuppose common good as a horizon of ethical deliberation.

If the ethical aim of any research is to increase the amount of common good or impact, then what is the ethical aim of the research evaluation? Or to paraphrase the latter question, whom the research evaluator is responsible to? An abstract notion such as common good serves as a horizon, which provides a both thematic and problematic framework for the ethical consideration in the research evaluation. Also Social Sciences and Humanities contribute to the common good, notwithstanding they are usually difficult to trace, track, and measure. Therefore, in order to make it operational ‘common good’ is to be divided into smaller realms inhibited by different collective stakeholders. In practice it means the following: every evaluator should consider the target groups (stakeholders) that research is dealing with. The authors of ENRESSH Policy Brief on Research Evaluation (Ochsner et al., 2020) claim that one can find four major categories—research production, research consumption and use, research policy and administration, evaluation services—and three intermediary categories of stakeholders, resulting in a taxonomy of twenty different types of stakeholders: from researchers to business, from cultural institutions to research councils, from taxpayers to learned societies, from funders to data providers, and so on. And this diversity of potential stakeholders must be taken into consideration when ethical issues are discussed.

Evaluation procedures should start with acknowledging responsibility to the disciplinary and academic communities. Only truthful research of high quality might be of any societal value. Thus, at the initial stage of an evaluation scientific integrity is needed to be checked and evaluated. Research with considerable flaws is incapable to benefit the collective good in any meaningful sense. If and only if the research under consideration is both epistemologically and methodologically plausible and valid, it has potential to benefit the broader amount of common good in extra-academic communities or the larger society as a means of practical problem solving. Therefore, setting the ethical principles for the research evaluation we must concentrate on these, which increase the amount of common good for different groups of stakeholders.

Schwandt (2015) urges general guidelines for evaluation grounded on the requirement of ‘critical thinking’, which involves the absence of political, personal, cultural and disciplinary biases and refuses the group-centered perspective and prejudices. More recently, Schwandt (2018) urges the necessity of a ‘professionalism in evaluation’, that is an ethical culture of evaluation, which concerns interpersonal relationships, the right conduct of evaluative process, the social responsibility, the necessity to serve the common good, to respect the dignity and cultural values of individuals and groups. Furner (2014) has suggested a conceptual framework for bibliometric ethics with the following essential tasks:

The different systems of assessment can affect scientific production in academic and research institutions (Whitley 2011) and the crucial point concerns the potential limit, due to the fear of the assessment, of the universities’ independence in pursuing research that follows unorthodox methodologies, or in developing innovative fields of research in disagreement with dominant approaches. The assessment of research activities, either ex ante or ex post, entail very important ethical issues. Pursuing the ‘common good’ in the research evaluation means that in case of multidisciplinary or interdisciplinary research every stakeholder with which the research is dealing with must be considered (ESF 2011). In this article, we consider three ethical theories—deontological, consequentialist and virtue ethics—and propose a mixed approach for developing a framework in the design and development of research evaluation. Moreover, the ethical theories can be deployed in analysing empirical findings for understanding the ethical approaches, as well as ethical dilemmas, in research evaluation.