Many scientific information scientists and research assessors may connect a citation index of scientific literature with Eugene Garfield’s vision of a multi-disciplinary core set of scientific journals selected on the basis of their citation impact, covering the best journals in science and forming the basis of his Science Citation Index (SCI), a scientific literature database launched in 1963. Soon a practice emerged that used the SCI not only for literature retrieval but also for research assessment, under the assumption that the appearance of a journal, scientific author or institution in the index can be interpreted as a sign of research quality. On many occasions, Garfield warned against over-interpretation and misuse of citation-based indicators in research assessment.

Scopus diverges from Garfield’s original model, as citation impact is not the only journal selection criterion. Table 1 provides insight into the national orientation and citation impact of journals covered. An Index of National Orientation (symbol INO-P) is defined as the percentage of articles published by authors from the country accounting for the largest number of articles published in that journal. The table shows that the percentage of nationally oriented journals (INO-P > 80) indexed in 2019 in Scopus is around 23 percent.3

The last column in Table 1 relates to citation impact, as expressed by Journal Impact Factor (JIF3). It shows that the percentage of journals for which JIF3 is smaller than 0.1, relative to the total number of journals, amounts to 7 percent. Raising the JIF3 threshold from 0.1 to 0.2, this percentage doubles. It is assumed that a JIF3 below 0.1 is extremely low for any journal, regardless of the subject field it covers.4

The effect that the inclusion of nationally oriented and/or low-impact journals may have upon citation-based author metrics depends upon the type of indicators calculated. One can distinguish two main types that are sometimes denoted as size dependent and size independent or, in terms of the key statistic calculated, as average based and sum based. A third type includes hybrid indicators, which combine elements from the size dependent and size independent approaches. Table 2 gives typical examples from these types and presents characteristic quotes of authors defending a particular type. The next section argues that the approach adopted by Ioannides et al. is essentially size dependent, based as it is on size dependent indicators and hybrid ones positively correlating with size dependent measures.5

Numerous experiences collected in the past decennia with the calculation of bibliometric indicators at the level of individuals have shown that the identification of all publications of a given individual researcher in a scientific literature database is highly sensitive to errors. The most important source of error is the occurrence of homonyms—different people with the same name, e.g., Smith, Jones, Lee, Liu, Andersen—and synonyms—different names for the same person, for instance, due to differences between full first name and nicknames, mixing up first name and family name, different transliterations of Cyrillic and other non-Latin names and name changing if a person assumes the name of a partner.

Although Ioannidis-2016 states that “Scopus author IDs were used for all author-based analyses,” they do not provide any information on how these Scopus IDs are created. Ioannidis-2020 refers to an article by Baas et al. (2020) describing how author profiles are created in Scopus. This information is the same as that given in Ioannidis-2019. Although Baas et al. (2020) do not give details on the author-clustering routine that underlies the author profiling and its ownership, they indicate three sources through which curation of these can be achieved: via ORCID, via the Scopus Author Feedback Wizard and via a special commercial Elsevier service.6

However, the AMD does not contain an indication as to whether the publications assigned to a particular author were actually verified by the person represented by this author. Hence, it is unknown how many author clusters included in the AMD are actually verified. Both this result and the lack of information about clustering software substantially reduce the transparency of the data included in the AMD.

The AMD indicates for each author an institutional affiliation, derived from an author’s most recent publication. For instance, an author with a 30-year career at University A who moved in the last year to University B (and who indicated his new affiliation in an article published in this year) is assigned in the AMD to B, not to A. As a result, an analysis of an institution based on the authors linked to it in the AMD may at best provide an indication of the past performance of the academic staff currently employed at that institution but does not necessarily give an impression of how the research staff appointed at—and in most cases funded by—an institution has collectively performed over the years.

The current paper focuses on a composite indicator that plays a key role in the inclusion of authors in the AMD and their ranking. It is presented in Figure 1. Its symbol is c and is calculated for each author in the AMD. It is defined as the sum of six components, each of which is basically calculated as the ratio of a specific citation indicator for a particular author to the maximum value of this indicator across all authors in the AMD. Rather than using straight counts, logarithmic values are calculated for the indicator value (plus one) both in a component’s numerator and in its denominator. Logarithmic values were used as the underlying citation distributions across authors are very skewed, a phenomenon that is clearly illustrated in Table 3 below.

The citation indicator in the first component, NC, counts the total number of citations in a given year to all publications by a particular author. Components 4, 5 and 6 take into account the number of co-authors in a given author’s papers or his or her position in the author list. NCS is based on citations to papers on which a given author is the sole author (single-authored papers) and NCSF on papers on which he or she is either single or first author, while NCSFL counts citations to single-, first- or last-authored articles. The second component is the H Index (Hirsch, 2005), and the third its variant, the Hm index (Schreiber, 2008).

One may argue in favor of the Composite Indicator that an indicator based on all of an author’s publications does not reveal well how his or her single-author papers are performing. However, the number of single-authored articles and their citations (NCS) is included in the count of single- and first-authored papers and citations (NCSF), which in turn are included in the count based on single-, first- and last-authored publications (NCSFL) and on the total number of citations (NC). As a result, the numbers of citations to an author’s various groups of papers are statistically dependent. This is clearly illustrated in the correlation matrix in Table 3. This table also shows that not only these elements of the Composite Indicator are statistically dependent. Pearson R values for correlations among indicators NC, H, Hm and NCSFL are all above 0.5 and range between 0.57 (NC versus Hm) and 0.92 (NC versus H). These four indicators all show R values above 0.4, with the total number of publications (NP) revealing that they are all size dependent.

As in Table 1 in Ioannidis-2016, all values in the correlation analysis in Table 3 are log transformed. Moreover, Table 1 is based on citations in the year 2019, obtained from Table-S7-singleyr-2019 in Ioannides-2020, while Table 1 in Ioannides-2016 is based on citations in a single year as well, namely the year 2013. In this way, the tables are both based on single-year studies and can therefore be compared. Generally speaking, Pearson R values in Table 3 are much higher than they are in the corresponding table in Ioannides-2016. This underlines the statistical dependence between the elements in the Composite Indicator.7

Allowing users to assess distinct categories of papers makes sense, even though it is argued below that indicators based on author sequences have a limited validity. In addition, Ioannidis-2016 states that equal weights were given to all six log-transformed indicators included in the composite for parsimony reasons and that “if, for whatever reason, one or more of these indicators are considered more essential in a particular field, one can weigh them more compared with the others.” However, it is questionable whether this consideration provides sufficiently valid grounds for including statistics for series of partially overlapping sets in a composite indicator that plays such an important role in the AMD. After all, it is the measure on which authors are ranked and is used to expand the AMD beyond the set of the top 100,000 authors.

All six indicators included in the composite measure are log transformed and standardized. Ioannidis-2016 argue that “log-transformations ensure that there are no major outlier values.” Their standardization method gives a value of 1 to the author with the highest raw value for a particular indicator. Ioannidis-2020 rightly underlines that “comparisons of citation metrics are more meaningful when done within the same subdiscipline.” However, their standardization method uses the highest raw value across all subject fields, while there are good reasons to use subject field-dependent highest raw values.

As expected, each indicator reveals substantial differences in these maximum values across subject fields. Calculating for each author a new composite measure based on maximum values per subject field using the Science Metrix classification into 174 subfields and correlating it with the original measure included in the AMD, the two composite indicators show a Pearson correlation of 0.77. Using the Science-Metrix classification into 20 main fields Pearson’s R amounts to 0.86. These outcomes show that applying a field-normalized standardization factor rather than one single factor across all subject fields does make a difference.8

The underlying basic assumption of the AMD is that one can derive an indication of the contribution an author has made to a multi-authored paper from the paper’s author sequence. The indicators in the AMD seem to be based on the assumption that in a multi-authored paper, the first and the last authors make the largest contribution to the paper. Indeed, there is evidence that research groups in experimental fields such as Physics and Chemistry often adopt an authoring practice according to which the first author is the PhD student conducting the experiment and the last author the supervisor responsible for—and often the intellectual owner of—the research program in which the PhD project is included. However, three essential limitations should be underlined:

Ioannidis-2020 states that in a first step, the top 100,000 authors are selected across all subject fields based on the Composite Indicator. In a second step, this set is complemented with authors not among the top 100,000 but still among the top 2 percent of their main subject field and publishing at least five papers. Although Ioannides et al. (2020) put the Composite Indicator into perspective by underlining that different components may be included or that different weights may be assigned to an indicator, it is clear that the Composite Indicator as defined in Figure 1 above plays the key role in deciding whether or not a particular author is included in the AMD. Hence, one should realize that analyzing the AMD and experimenting with the selection of indicators and weights can only be applied to those authors who are already in the AMD and can therefore not be used, for instance, to examine the effect of changes in the formula of the Composite Indicator upon the inclusion of authors in the AMD. It follows that in the assessment of an individual author, an evaluator cannot simply assume that one is making a correct judgment if it is based on whether or not an author is included in the AMD.

One type of use of the AMD in the assessment of an individual researcher, for instance, for hiring or promotion purposes, is to look up the author entry in the AMD with the same name as the assessed researcher. Next, an assessment criterion is defined, for instance, being included in the AMD or having a Composite Indicator score in the top quartile of this indicator’s distribution. Finally, a decision is made purely on the basis of the thus-obtained outcome, without taking into account any other sources of information. This type of use can be denoted as desktop bibliometrics. The creators of the AMD make clear that they are strongly opposed to this type of use. So is the current author (Moed, 2017, 2020). Judgment of an individual’s performance by applying assessment criteria based on thresholds for a particular bibliometric indicator is indefensible not only if the validity of the indicator is questionable but also if threshold values themselves are not well founded.

The AMD creators rightly point out that multiple co-authorship is a rule rather than an exception, especially in the natural and life sciences. As a consequence, publications (co-)authored by an individual researcher are often, if not always, the result of research to which other scientists have contributed as well, sometimes even dozens of them. The crucial issue is how one should relate the citation impact of a team’s papers to the performance of an individual working in that team. It is fully appropriate that the creators of the AMD dedicate so much attention to this issue. However, one must realize that performance of an individual and the citation impact of his papers relate to two distinct analytical levels.

The current author defends the position that a valid assessment of the research performance of individuals can be properly made only on the basis of sufficient knowledge of the particular role they played in the research presented in their publications, for instance, whether this role has been leading, instrumental or technical. In addition, other manifestations of research performance should be taken into account as well. Calculating indicators at the level of an individual and claiming they measure by themselves an individual’s performance, statistically sophisticated as they may be, suggests an accuracy of measurement that cannot be justified. This is especially also true for the AMD Composite Indicator. Ultimately, its validity does not depend upon the number of components in the indicator or on the level of sophistication of their weights.

The very existence of a database with “top” researchers invites evaluators and other interested users to use the information for their own evaluative purposes. The AMD creators explicitly refer to the entities analyzed in the AMD as scientists, not as authors, thus emphasizing the personal rather than the statistical nature of the data. As outlined above, for part of the 165,000 authors, the publication lists have to some extent been verified by the corresponding scientist, but there is no information available on how large this fraction actually is. In addition, the AMD does not include for each author a flag indicating whether or not the underlying data were verified. The current author believes that the lack of this information seriously hampers the transparency of the AMD that such a flag must be included.

What is more, it would have been much more appropriate to include only scientists whose algorithmically generated publication lists were verified and who explicitly have given their consent. The fact that the statistical de-duplication of author names and assignment of documents has already taken place in Scopus and has not been contested by subjected scientists does not justify the creation of the AMD in its current form, as Scopus is primarily a scientific literature search tool in which author names are content descriptors, not scientists subjected to a performance evaluation, many of whom may not even be aware that they are included in the AMD.

The analysis of Scopus content coverage revealed that this database indexes a substantial number of nationally oriented journals with a low citation impact. Although there is evidence that once they are indexed in Scopus, many of these journals internationalize and increase their citation impact (Moed et al., 2021), their inclusion may distort size independent, average-based citation indicators. In terms of the distinction between indicator types made above, the decision made by Ioannidis et al. (2016; 2020) to apply size dependent or hybrid indicators is well defensible. However, this choice has its limits.

The AMD focuses almost exclusively on senior scientists. Early career scientists and members of emerging research groups who will shape science and scholarship in the near future hardly appear in the AMD. A rough indication of the extent to which early career researchers (ECRs) are covered in the AMD can perhaps be based on the assumption that an ECR would publish only papers as a first or single author, and not yet publications as a last author. There appear to be only about 1,000 authors meeting this criterion, accounting for 0.6 percent of the total number of AMD authors. The AMD aims to cover “top” researchers. The Composite Indicator and its components are size dependent and strongly biased in favor of senior authors with long scientific careers.11 The current author wishes to underline that this size dependence is a choice made by the creators themselves. In the field of bibliometrics, there is also a line of development of size independent or “relative” indicators.12

It was argued above that the issue as to how one should deal in research assessment with multi-authored papers can to some extent be enlightened by bibliometric indicators but cannot be solved bibliometrically. It was concluded that it is problematic to justify an evaluative judgment merely based on quantitative indicators and threshold, as they suggest a false precision. The following question then arises: how can the AMD be used in a proper manner? One could argue that the data on authors presented in the AMD represent only a first step in an assessment and that additional bibliometric data can be retrieved from other data sources, especially the online version of Scopus and the special online bibliometric tool SciVal created by Elsevier. Although the intentions of the creators of the AMD are beyond any doubt to calculate bibliometrically founded indicators at the author level available to a wide audience, this argument would underline that the AMD is de facto a promotion tool for these two Elsevier products.

The creators of the AMD have indeed made an important step toward a bibliometric assessment tool by creating a rich intermediary dataset with bibliometric indicators, open for further analysis to all interested people. Although the Composite Indicator is the key measure, its components can be used as separate indicators as well. In addition, the AMD contains other interesting features not discussed in the current paper, such as the possibility to analyze citation counts including or excluding author self-citations.

Technically, it seems feasible to add in a follow-up version of the AMD the verification status of the publication data relating to a particular author, as this information is available in the Scopus system, or to include only authors who have validated their data. However, making the database interesting for ECRs by increasing the number of included authors and adding size independent indicators seems hardly doable within the framework of the current AMD model. The current author would like to broaden the perspective and bring in three new elements that could play a role in the further development of AMDs.

Firstly, the AMD is perhaps still too much based on more classical data-handling approaches developed during the past decades and does not yet fully profit from tools to create interactive and flexible bottom-up applications enabling interested users to go back to the raw data, decompose existing indicators and generate new, more fit-to-purpose measures if needed. Secondly, the key function of a new version could be to deliver bibliometric data informing an author self-assessment. It would enable a scientist to select and verify his or her own publication data; next, it creates a set of ‘candidate’ benchmark authors or groups using algorithms similar to those proposed by Eugene Garfield for evaluating faculty (Garfield, 1983a, 1983b). It may also offer a flexible benchmarking feature for users as the practical realization of Robert K. Merton’s notion of a reference group, i.e., the group they “do not necessarily belong but aspire to” (Merton, 1996).

Thirdly, it could also function as an educational tool to become more acquainted with the ins and outs of bibliometric indicators by making users aware of the technical and evaluative choices that have to be made in a bibliometric analysis. It could stimulate the user to specify at least some of the elements from an evaluative framework overarching the self-assessment, thus stimulating the user to reflect upon this framework. It would reveal to a user how outcomes of bibliometric assessment depend upon the way initial publication lists, author benchmark sets, author position weights and subject delimitations are being defined and upon the role of particular evaluative assumptions and setting of thresholds. It could contribute to the transparency of a research assessment process by enabling those subjected to an assessment in their external professional environment to critically follow this process and could defend them against inaccurate calculation, misinterpretation or inappropriate use of indicators.13

Obviously, research assessment is much more than just bibliometrics. Research assessment requires an overarching evaluative framework based on normative views on what constitutes research performance and which policy objectives should be achieved. Informetricians should comply in their scientific work with the methodological principle to maintain a neutral position toward an assessment’s constituent policy issues, the criteria specified in the evaluative framework and the goals and objectives of the assessed subject. As professional experts, their competence lies primarily in the development and application of analytical models given the established evaluative framework. They may contribute to a productive combination of qualitative and bibliometric tools. In addition, as more and more bibliometricians have been involved as actor, advisor or observer in actual assessment processes using bibliometric indicators, they can report on their experiences in these processes to a wide scholarly and policy audience.