When decision makers in science (scientists or managers) assess units (e.g., research groups or institutions), complex solutions are generally preferred: decisions rely on complex and time-consuming peer review processes. Since scientific quality is seen as multi-dimensional phenomenon, complex ways are regarded as a gold standard for research evaluation processes. In a recent paper, Bornmann and Marewski (2019) argue that in certain evaluation situations bibliometrics-based heuristics (BBHs) might be good alternatives to complex assessment processes. BBHs are reasonable decision strategies which reduce the considered information in assessments to indicators based on publication and citation data. In this short communication, main lines proposed by Bornmann and Marewski (2019) are summarized in a brief overview.

Bibliometrics is a method for assessing research based on publications and their citations, which is increasingly used in various research evaluation processes (alongside and in combination with peer review, Moed, 2005, 2017). The popularity of the method can be traced back to the following advantages:

In recent years, empirical studies have shown that bibliometrics come to similar results as reviewers in various research evaluation situations. Expensive and time-consuming peer review processes are used in many evaluations. One of these complex processes is the UK Research Excellence Framework (REF). In a recent study, Pride and Knoth (2018) investigated the correlation between bibliometrics (citations in Microsoft Academic) and peer review (grade point average ranking). The study is based on nearly 200,000 papers from 36 disciplines submitted to the REF 2014. The authors found that the citations of the papers are strongly correlated with peer review assessments at the level of institutions and disciplines. The authors also demonstrated that the citation data can be used to predict institutional rankings in certain disciplines that are based on peer review assessments. The high degree of prediction accuracy of bibliometrics found in this study questions the use of the expensive and time-consuming peer review processes.

The study by Pride and Knoth (2018) is not an isolated case: high correlations between bibliometrics and peer review have not only been reported for the REF (e.g., Harzing, 2017; Traag & Waltman, 2019), but also for other peer review processes (e.g., Auspurg, Diekmann, Hinz, & Näf, 2015; Diekmann, Näf, & Schubiger, 2012). The results of these studies indicate that a focus on bibliometrics might not be a limitation in decision making. Simple decision making strategies based on bibliometrics can be similarly accurate to complex strategies, such as peer review processes. Bornmann and Marewski (2019) recommend that assessment processes in science be empirically studied to reveal the situations in which this is probably the case and those in which it is not.

The investigation of the relation between bibliometrics and peer review is particularly interesting in informed peer review processes. These are evaluation processes in which decision makers are informed by certain indicators. How do the decision makers use these indicators and how do they rely on them? Could indicators replace the decision makers’ judgments (partly or completely)? Do decision makers simply count the number of papers in order to assess researchers? Do they take the Journal Impact Factor (JIF) of these papers - one of the most popular journal metrics - into account? Do they prefer to rely on the h index for their decisions? Bornmann and Marewski (2019) suggest that these and similar questions can be empirically answered by studying the use of bibliometrics within the heuristics research program.

The conceptualization and study of BBHs go back to the heuristics research program introduced by Gigerenzer et al. (1999). Many decision makers apply fast-and-frugal heuristics (simple decision strategies or rules of thumb) in the process of making judgments (see, e.g., Astebro & Elhedhli, 2006). Research from the area of judgement and decision making reveals that decisions based on a small amount of information can come to similar or better results than decisions made using a large amount of information (Gigerenzer & Gaissmaier, 2011). Bornmann and Marewski (2019) assume that this might be also the case for decisions in research evaluation. The authors conceptualize the use of bibliometrics in research evaluation based on the heuristics research program and introduce the term BBH. Reasonable BBHs might be able to significantly reduce the effort and time involved in decision making in research evaluation.

Research on BBHs is not interested in whether certain bibliometric indicators are mathematically correct; rather, the purpose of this research is to reveal decision situations in which the application of bibliometrics leads to appropriate judgements. The heuristics research program is rooted in the bounded rationality approach by Simon (1956, 1990). The author argues that people prefer simple decision strategies when resources are scarce (e.g., cognitive capacity or time) and when many decisions are required within a short time. Through the use of these strategies, people are ecologically oriented in decision making. The unbounded rationality approach forms the contrast to the bounded rationality approach. It assumes that people focus on complete information. The approach is logically oriented: various criteria are applied to weight many pieces of information and combine them to reach an ‘optimal’ decision. The unbounded rationality approach assumes unlimited time, complete knowledge, and vast information-processing capacity. Since these assumptions are seldom fulfilled in typical research evaluation processes, the bounded rationality approach seems more realistic.

Although the application of fast-and-frugal heuristics in research evaluation has never been investigated, results from other fields are available. These results may suggest that fast-and-frugal heuristics are also relevant for the research evaluation field. The overviews of Marewski, Schooler, and Gigerenzer (2010) and Katsikopoulos (2011) show, for instance, that heuristics are able to contextualize and inform diagnosis in medicine.

Bornmann and Marewski (2019) argue that adapting the heuristics research program (Gigerenzer et al., 1999) to evaluative bibliometrics would be an important step towards the conceptualization of the evaluative use of bibliometric indicators. The introduction of bibliometrics-based social heuristics, as described in the next section, may serve as an example for such a conceptualization. The section demonstrates the reasons that can be given for the use of citations for evaluative purposes. Other examples for conceptualizations can be found in Bornmann and Marewski (2019), Bornmann (2019), and Bornmann (2020). For example, Bornmann and Marewski (2019) focus on one-reason decision making heuristics, which exclusively base decisions on just one ‘clever’ cue without considering any other information. This cue can be a certain bibliometric indicator such as the number of papers belonging to the 10% most frequently cited papers in their field and publication year.

When classifying objects (e.g., papers) according to their ‘quality’, social information and thus social heuristics might come into play - and be useful from an ecological point of view. As Gigerenzer and Gaissmaier (2011) point out, “social heuristics prove particularly helpful in situations in which the actor has little knowledge” (p. 472). Why and when can social information be useful in science evaluation? Bornmann and Marx (2014) discuss the benefits of citation analysis in research evaluation based on what Galton (1907) calls the ‘wisdom of crowds’. He reports on the results “of a contest at an English livestock show where contestants were asked to guess the weight of a publicly displayed ox. After sorting the 787 entries by weight, Galton found that the median estimate of 1,207 pounds differed from the true weight of 1,198 pounds by less than 1%” (Herron, 2012, p. 2278). When many people make a judgement or an estimate, one can expect that it will be valid (see also Laland, 2001). Averaging the judgments of others has been proposed as social heuristics to exploit the ‘wisdom of crowds’ (Hertwig & Herzog, 2009).

It is the approach of citation analysis that it is based on the judgements of many others: a publication is frequently cited by the scientific community when it has turned out to be useful. The wisdom of crowds in bibliometrics usually leads to skewed citation distributions: only a few publications receive many citations and most of the publications only a few or no citations at all. Only a few publications seem to be extremely useful in various fields, research contexts, and time periods in promoting science and development (Bornmann, de Moya Anegon, & Leydesdorff, 2010). Most publications do not seem to be of significant importance for later research. In other words, social citation data point to an unequal distribution of usefulness and recognition of research. Based on the conceptualization of citations as bibliometrics-based social heuristic, it might be ecologically rational to rely on citation counts to single out very useful papers.

Following Raab and Gigerenzer (2015), Bornmann and Marewski (2019) define three goals for investigating BBHs within the heuristics research program:

Many studies, which have hitherto empirically investigated bibliometrics, focus on certain flawed bibliometric indicators (e.g., the h index). Authors of these studies have proposed to replace flawed indicators by more suitable indicators. For example, at least 50 variants of the h index have been introduced since 2005 (Gasparyan et al., 2018). Only a few studies in scientometrics have investigated the use of bibliometrics in specific research evaluation situations (e.g., Hammarfelt, Rushforth, & de Rijcke, 2020; Moed, Burger, Frankfort, & van Raan, 1985; van den Besselaar & Sandström, 2020). The study by van den Besselaar and Sandström (2020) also deals with ways of handling contradicting bibliometric information (e.g., based on different literature databases), and decisions why and when to deviate from the bibliometric information.

All these studies on (the use of) bibliometrics are necessary, but have not been undertaken in the context of a coordinated research program. Bibliometric research lacks a common program that could function as a framework for empirical research. The heuristics research program can serve as such a framework, so that the usage of bibliometrics can be conceptually understood, empirically studied, and effectively taught (Bornmann & Marewski, 2019).

If one assumes that researchers apply bibliometric shortcuts (BBHs) to reach decisions in research evaluation, one should investigate (1) how researchers use these shortcuts (BBHs) and (2) how successful they are in various evaluative situations. According to Mousavi and Gigerenzer (2017), “heuristics per se are neither good nor bad. They can be evaluated only with respect to the environment in which they are used. The more functional the match is between a heuristic and the environment, the higher the degree of ecological rationality of the heuristic. The functionality of this match is verifiable by the extent of its success rather than fulfillment of coherence requirements” (p. 367). Since evaluative processes and the use of bibliometric indicators in these processes change continuously, evaluation practices should be continuously empirically studied. For the appropriate evaluation of heuristics, Marewski et al. (2010) recommend that these principles are followed: (i) specify precise formal heuristics models that can be tested; (ii) test heuristics comparatively to identify better heuristics; (iii) test heuristics in new situations, and (iv) test heuristics in the real life settings.