A well-known example of the study of values is the work by Robert K. Merton on social norms in scientific-scholarly research communities, conceived by him as ideals that are dictated by the goals and methods of science and are binding on scientists (Merton, 1942; 1979).

A typical example from the field of quantitative science studies is empirical research into how researchers perceive “research quality”. This type of research does not aim at drawing conclusions as to whether such perceptions are valid or not, but at examining how researchers form quality judgments and which decisions they make in a particular assessment process. One may also investigate the empirical conditions under which certain quality perceptions emerge or under which one perception is more likely to be shared than another. Research on biases in peer review is a good example of this type of investigation. A research analyst may find that females are underrepresented in permanent academic staff of a particular university. Such a conclusion should be empirically and statistically justified, independently of whether the analyst finds discriminating against females in promotion processes unfair.

In scientific-scholarly research practice, at the level of theoretical-empirical statements about objects, value judgments may enter, reflecting the valuation or appreciation of the analyst toward the object. Such personal judgments can be mistakenly conceived as representing knowledge of the object validated in the research itself. A typical example is the use of the term ‘closed access’ to indicate subscription-based access to scientific journals. Additionally, the use of the term ‘peripheral journal’—even if it has a well-defined meaning in network analysis—may give rise to a negative impression of a journal among non-experts.

The methodological requirement of value neutrality at the level of statements on objects based on empirical research dictates that such personal valuations of the object of research should be avoided. A researcher who wants to make them known to the reader should make clear that the qualifications are not based on the research itself, but are a priori assumptions of the investigator. The requirement of value neutrality also has the following implication: it is methodologically incorrect to neglect, on the basis of a value interest, relevant facts that fall within the intended field of research but contradict a preconceived conclusion and to one-sidedly emphasize facts that do support it.

This section is not about values as research object, nor about scientifically unfounded or hidden prescriptions and valuations by the analyst, but about the value basis on which science itself operates, including rules that standardize and direct scientific activity. These values can be explicated and discussed at a level higher than that on which scientific statements are made. It can be denoted as a meta-level.

The principle of value neutrality is itself a normative requirement that should be analytically positioned at this meta-level. Although the methodological rules at this level do have an ethical component, it is not their ethical dimension that is at stake in this section. These rules are constitutive for scientific inquiry. It is not so much that researchers who violate these rules are doing ‘bad science’, but rather that they are not doing science at all, in the same way as, for instance, two chess players who allow pawns to move three steps forward do not play chess.

The principle of value neutrality has strong implications for the position of an informetric analyst contributing to an assessment process toward the policy objectives underlying the process and the evaluation criteria applied in it. As stated by Moed, “A basic notion holds that from what is cannot be inferred what ought to be. Evaluation criteria and policy objectives are not informetrically demonstrable values. Of course, empirical informetric research may study quality perceptions, user satisfaction, the acceptability of policy objectives, or effects of particular policies, but they cannot provide a foundation of the validity of the quality criteria or the appropriateness of policy objectives. Informetricians should maintain in their informetric work a neutral position towards these values” (Moed, 2017, p. x).

The methodological rules stated at the meta-level constitute scientific practice and have an internal-scientific character. However, the methodological principle of value neutrality does not preclude that also political, moral and cultural values have a normative function toward scientific-scholarly research. Following Duintjer (1974), the next six sections show, from different viewpoints, the role of the social-political-cultural context in which research is being carried out.

Duintjer underlines that each problem statement is guided by selective points of view that are incorporated in theoretical core concepts and that determine the selection of empirical data. Extra-scientific values often hide behind these selective viewpoints and core concepts. “The fact that scientific research is guided by selective points of view, which in turn are inspired by social values, is not formally the same as a normative setting of these values. In research, these values can be assigned a hypothetical status” (Duintjer, 1970, p. 29).

Against the dependence on extra-scientific values, he recommends taking the following precautions: “making the concealed or implicit value background of selective viewpoints explicit and recognition of its non-scientific character; insight in the limits of the empirical knowledge acquired within such points of view; receptivity to investigations adopting different or wider viewpoints” (Duintjer, 1970, p. 32).

In quantitative science studies and applied evaluative informetrics, the concept of research impact plays a key role. In many studies, a tacit assumption seems to be that generating impact is per se valuable, and that it is worthwhile being studied in bibliometric studies and being assessed in research evaluation. However, the validity of this assumption cannot be grounded in informetric or sociological research, although it determines which data are to be collected and analyzed (namely, citation data). Impact as a value should be given a hypothetical status.

A second example relates to the distinction between formative and summative evaluation. In summative evaluation, the focus is on the outcome of a particular activity. This outcome can, for instance, be used in a funding decision. Formative evaluation assesses the development in an activity at a particular time and focuses on improving its performance. Whether an assessment process should be formative or summative cannot be decided on the basis of informetric grounds.

In empirical research into the effects of the application of indicators upon researchers’ behavior, the analyst’s view on whether this application is valuable and whether or not citations indicate performance should not play a role in the justification of empirical statements. However, this view may influence the selective viewpoints adopted during the set-up of the study. The principle of value neutrality does not imply that such influence is inadmissible, but rather that it should be made explicit in the presentation of the study set-up and discussion of the outcomes.

As Duintjer states, “The choice of research fields in science is not only related to conceptual points of view, but also to the extent to which the expected results of the research can foreseeably be of importance for certain social power groups, which are sometimes involved in conflicts and who will use these results in their favor and to the detriment of their opponents” (p. 32). Conversely, value decisions are also at stake when investigations are launched to unmask ideological facade decorations and hidden power relations and can therefore foreseeably contribute to public awareness or political “Aufklärung” (p. 33).

For instance, managers at a research institution or research funding organization may commission informetric researchers to conduct a study with the aim of using its outcomes to solve an internal conflict or to enhance the status of their organization. They may even commission informetricians to develop a strategy to artificially increase the bibliometric indicator score of their institution, with the objective to enhance its position in world university rankings. This type of applied evaluative informetrics is directly serving certain vested interests.

However, critical informetric studies can also question a particular situation or practice. For instance, the work of the Leiden team headed by Anthony van Raan in the early 1980s strongly underlined the critical role of bibliometric indicators and their potential to raise critical questions to academic department managers and funding organizations about their quality assessment and funding policies. In a study of research groups in their university, they obtained evidence of conservatism in funding policies, favoring groups with a long publication and funding track record, at the expense of emerging groups headed by young researchers exploring new approaches (Moed et al., 1985).

“Scientific explanations and predictions are often sought with the help of experiments, looking for behaviors of an isolated class of phenomena under self-determined conditions. But when organizing an experiment with laboratory animals or test subjects, one also has to deal with moral values that regulates also outside of science our relationship to animals and humans” (Duintjer, p. 34).

When developing and testing new assessment methodologies or indicators, their validity can be examined only if they are applied to “real” performing entities, such as individual researchers, groups or institutions. Therefore, application experiments or ‘try outs’ are organized, in which the outcomes of a method are discussed with the subjects of the assessment. However, if these outcomes become public—for instance, published in a journal article—or are shared with those who commissioned the experiment—e.g., a managing director of the commissioning organization—they may harm the prestige of the subjects under assessment, even if the final conclusion holds that the experiment has been only partially successful.

The environment in which evaluative-informetric experiments take place is not a scientific testing ground. Their effects upon the outside world may be irreversible. The experimental setting has not only a theoretical-methodological dimension but also a moral one: how to protect subjects under assessment against loss of status during and after a bibliometric experiment.

Duintjer emphasizes a so-called “structural equivalence” of science and technology. All theoretical statements or explanations in modern science can be converted into technological statements that answer the question “what could be done?” “All pure theoretical research in modern science provides society with means to steer nature and man, without indicating in which direction one should steer” (p. 38). Duintjer advocates an “overall, democratic value discussion regarding the direction, goals and standards to which society directs itself and is directed” (p. 38). “Obviously, not merely scientists should decide on the direction of the whole of society, especially not because value judgments cannot be derived from theoretical-empirical knowledge itself” (p. 38).

As outlined in the introduction section, Moed (2017) distinguished four levels of intellectual activity in a research assessment process: policy and management, evaluation, analytics and data collection. On the one hand, he stated that a view of what is valuable in science and scholarship and its corresponding evaluation criteria and standards cannot be grounded in empirical research conducted at the analytics level. In this respect, evaluative informetrics is and should be value neutral. On the other hand, he argued that “in the proper use of informetric tools an evaluative framework and an assessment model are indispensable. To the extent that in a practical application an evaluative framework is absent or implicit, there is a vacuum, that may be easily filled either with ad-hoc arguments of evaluators and policy makers, or with un-reflected assumptions underlying informetric tools” (Moed, 2017, p. xi).

Following Duintjer’s statements quoted above, one can maintain first of all that not merely informetricians should decide on the values, standards and political objectives underlying an assessment, especially not because value judgments cannot be derived from informetric research itself. What is more, one can argue that an overall discussion is needed within the academic community and the research policy domain about the objectives and criteria in research assessment processes at various levels of aggregation—e.g., individuals, institutions—and about appropriate conditions for the use of informetric indicators in these processes.

Duintjer introduces this interface between science and values as follows: “As already said, the theoretical-empirical knowledge of science does contain technological information about what we can do, but it does not yet answer the normative question of what we should do. The latter question was precisely the subject of the advocated value discussion (see previous section, HM). One may wonder, however, whether one can perhaps derive from the ethos that underlies scientific practice also elements that can be presented to society as moral and political values” (p. 39/40).

He proposes a possible extension not only of norms regulating the scientific attitude toward colleague-researchers but also of methodological rules constituting a researcher’s relationship with the object of research. He refers to, among other authors, Karl Popper (1966) and Hans Albert (1965), who, building upon the tradition of ‘critical rationalism’, propose that a moral belief in reason as a method to solve problems in science could function as a norm also in society.

Within the context of the current paper, the question emerges as to whether and how science-internal norms could provide guidance in science assessments, especially in the further theoretical and practical development of evaluative frameworks. Following Duintjer’s line of reasoning, one could, for instance, propose to extend the intra-scientific principles of openness and adopting a critical attitude, or the ‘Mertonian’ norm of disinterestedness toward the formation of an evaluative judgment in a research assessment process (Biagetti and Gedutis, 2019).

As outlined in Section 3, scientific statements and practices are regulated by specific values in the form of objectives and rules. According to Duintjer, these values define what could be termed as a particular access road into reality. The value field of modern science is not the only value sphere, but rather forms part of a comprehensive whole of values next to other value spheres, such as those of politics, art, philosophy of life and existential experience.

In research assessment, distinct value domains come together, making it a complex activity that is difficult to grasp. The term ‘research performance’ relates to an agglomerate of internal-scientific values, such as ‘methodological soundness’, or notions as complex as ‘contribution to scientific-scholarly knowledge in a discipline’ or, moving outside the boundaries of a particular discipline, ‘to the advancement of scientific-scholarly knowledge in general’. Reaching beyond the domain of science and scholarship, values in research assessment may refer to ‘improving the state of mankind’ or to technological, economical or societal merit.

The domain that Duintjer denoted as “modern science” and that is outlined in the introduction section does not embrace other forms of scholarship that he denotes as “hermeneutic”3—philosophy is a typical example—and “socio-critical”.4 It is good practice to denote these disciplines as ‘scholarly’ rather than ‘scientific’. The current author defends the position that these domains of scholarship could contribute to the formation of an overarching evaluative framework mentioned above, in which they could have an enlightening rather than a technological function. Such contributions can be marked as scholarly but fall outside the value area of modern science and indeed cover “other parts of a wider life world”. However, it should be noted that there is no explicit reference to this interpretation in Duintjer’s article.