One of the strongest advantages of Mendeley readership data is its free access via the Mendeley public API4 or via manual collection of readership data from the Mendeley catalog.5 The Mendeley API offers free access to diverse metadata and endpoints including among others readership statistics, as well as breakdown statistics by users’ academic statuses, disciplines, and countries. The best approach to collect Mendeley readership data is to directly use the public API, particularly over other third-party sources (e.g. vial Almetric.com or PlumX, see Robinson-García, et al., 2014; Ortega, 2018; Zahedi and Costas, 2018). Another important advantage of the Mendeley API is that it allows for data collection based on different publication identifiers (DOI, PMID, arXiv ids, Scopus ids).

Among the limitations of Mendeley readership data we can mention the lack of access to temporal and longitudinal data (i.e. the readership history of publications), the short readership window for recent publications, the lack of options for bulk data download, and the dependency on using publication identifiers (DOI, arXiv ID, Scopus ID, PMID, etc.) for querying the API (Zahedi, Bowman, and Haustein, 2014; Haustein, 2016; Zahedi and Costas, 2018). Other known limitations of Mendeley data include the biases in coverage of some geographical areas (e.g. China or Russia) (Thelwall, & Maflahi, 2015; Fairclough, and Thelwall, 2015; Alperin, 2015; Zahedi and Costas, 2017); the potential manipulability of Mendeley indicators by registering duplicate records; the inaccuracies in the proper identification of Mendeley users (e.g. wrong academic status or geographic origin); or the data qualities issues in Mendeley (e.g. the limited availability and quality of bibliographic metadata, the existence of duplicate records with different publication identifiers, etc.). These limitations emphasize that it is often necessary to resort to other bibliographic data sources (e.g. WoS or Scopus) for advanced analysis (Wouters and Costas, 2012; Haustein, 2016; Wouters, Zahedi, and Costas, 2019; Thelwall, 2020) able to overcome the lack of metadata in Mendeley (we will illustrate this issue in this work).

This methods report is structured as follows: section 2 describes the methodological approach. Section 3 provides general overviews related to the publications, coverage and impact (readership and citations) related to the six different example universities. Section 4 provides focused examples on the practical uses of Mendeley readership, and finally section 5 provides and discussion of these practical use, and section 6 condenses some specific best practice recommendations of how Mendeley readership can be used for assessment purposes.

In this study we worked with bibliographic data from Crossref.6 The choice of Crossref as a bibliographic data source for this illustrative exercise is motivated by the need of having a large independent bibliographic database containing all sorts of publications, and not being restricted to the coverage criteria of WoS and Scopus (van Eck et al., 2018), or even Mendeley. The full Crossref database is stored in an in-house relational database available at CWTS. This database was downloaded in August 2018 through the public REST API. This in-house database has been used in order to collect the output of six sample universities (Leiden University, Stellenbosch University, Dalian Technology University, Sao Paulo University, University of Harvard, and Curtin University). The selection of these universities responds only to illustrative reasons (i.e. they are used as examples of the practical uses of readership indicators, thus there is no special meaning in their choice). We tried to select a university per world continent (i.e. Europe, Africa, Asia, South America, North America and Oceania) and their choice was also based on the personal knowledge of the authors of this report of some of them.

The universities were queried in the affiliation field of Crossref by using their English names. The authors’ affiliation information in the Crossref database is based on the Crossref member organizations and enrichment done by Crossref itself (Hendricks et al., 2020). No time restriction (limitation to publication dates/years) in collecting Crossref DOIs was applied, in order to illustrate how Mendeley readership can also be used to analyze publications from any time. It is important to highlight that the number of publications with affiliations in Crossref is not indicative of the true output of the universities selected, since Crossref does not index all affiliations of all publications recorded in Crossref.7 Therefore, we do not claim any completeness in our data collection, and the same analysis could have been done using other bibliometric databases (e.g. Scopus, WoS, Dimensions) that have more curated affiliation information. However, by choosing Crossref we want to emphasize the possibilities of Mendeley of providing readership indicators to publications available in an open and free database, without initially needing the use of a commercial and more curated database. A total of 79,416 different Digital Object Identifiers (DOIs) of publications identified in Crossref for the selected universities were collected.

Readership data used in this report were based on the annual Mendeley data collection carried out at CWTS of Mendeley data (annual data collection of 2018). In the annual CWTS data collection approach, the Mendeley API8 is queried based on yearly updated lists of DOIs (obtained from Crossref, Web of Science (WoS) or Scopus, among other data sources). The downloaded JSON files resulting from querying the Mendeley API are then parsed and stored as relational tables in a SQL database environment, which allows for a more advanced use of the original data (e.g. use of the breakdowns of types of users, disciplines, or countries).

For some of the analysis some additional metadata (e.g. citation counts, thematic classifications and document types) that were not available in either Crossref or Mendeley were necessary. In order to obtain these other metadata elements, the Crossref records were matched with the CWTS in-house databases of WoS and Scopus based on DOIs. WoS and Scopus citations counts were added to the Crossref records. Moreover, journal-based subject classifications (CWTS NOWT classification9), publication years, and document types of the Crossref records were also extracted from the WoS database in order to perform some of the analysis.10

The restrictions of the most common bibliometric databases (e.g. WoS or Scopus), with regards to their coverage of some fields, language, and publication formats, represent one of the most common challenges in scientometric studies (Torres-Salinas, Cabezas-Clavijo & Jimenez-Contreras, 2013; Van Raan, Van Leeuwen & Visser, 2011; Archambault & Larivière, 2006). In this section we illustrate how readership indicators offer practical opportunities for identifying the impact of publications that are not indexed in the most common bibliometrics databases.15

Figure 1 depicts the share of Crossref DOIs that are covered by the different data sources (WoS, Scopus, or Mendeley) or any combination of them (see also supplementary file 1. Table A1 in the appendix16). The number of overlapped DOIs across all data sources is 40,251 (51%). Overall, Mendeley has the largest coverage of publications that are not covered by any of WoS or Scopus databases (n = 18,658; 23.5% of the total publication dataset). Put differently, Mendeley offers metrics for a larger set of publications for which no metrics are available in WoS or Scopus. Share of publications covered both in Mendeley & Scopus alone (11,261; 14.2%) is higher than the share of publications covered both in Mendeley & WoS alone (5,331; 6.7%) and WoS & Scopus (273; 0.3%) (Figure 1; see also supplementary file 1. Table A1 in the appendix17). These results show that Mendeley can be used to identify the impact of publications not indexed in these two citation databases.

Document types like editorial materials, letters, news items, book reviews or meeting abstracts are types of publications that focus more on disseminating scientific debates, news, opinions, or summarized information, and typically receive relatively low numbers of citations. Due to their lower citation density they are usually deemed not suitable for robust citation analysis and are often excluded from citation analyses (Waltman et al., 2011).18 Previous studies have shown that these document types have some coverage on Mendeley, and they also have higher readership counts than citations (Zahedi and Haustein, 2018). Thus, it can be argued that the impact of these document types could be better assessed with readership indicators.

Since the classification of document types in Crossref has fundamental limitations (Visser, Eck, and Waltman, 2020), the WoS document type classification was used for the set of DOIs (n = 33,868) from the years 2012-2018.20 Figure 2 (and Supplementary file 2. Table A2 in the appendix21) presents the coverage and average values of the Crossref DOIs for both WoS & Scopus citations and readership for the different document types identified in WoS. While articles and reviews are the document types that are most cited in WoS and Scopus, data papers, reviews, articles, editorial material, letters, and news items are the most saved document types on Mendeley. These results support the idea that Mendeley readership can be used to identify the readership impact of these document types.22

Social sciences and humanities are among the research fields worst covered in citation databases (Nederhof, 2006). Their low citation density makes it more difficult to study the citation impact in these fields, as well as to compare their impact with other fields. However, Mendeley readership has been observed to have a higher density than citations in these fields (Thelwall, 2015; Thelwall, 2017b; Zahedi, 2018), thus opening the possibility for more reliable and substantial studies on the impact of social sciences and humanities. In this section we show how readership indicators can offer some impact evidence when studying these fields.

Since Crossref does not count with a comprehensive classification of all documents (Visser, Eck, and Waltman, 2020), the set of DOIs from the years 2012-2018 and also covered in WoS (n = 33,868) were classified into the seven main fields of science based on NOWT classification scheme23 available from CWTS in-house database. Citation and readership indicators from CWTS in-house WoS, Scopus, and Mendeley databases were calculated and aggregated by the seven NOWT24 main fields of science (Table 3).

The results show that in all fields, readership density exceeds citation density. On the one hand, Social and behavioral sciences publications exhibit the highest readership density across all fields. Publications from this field on average have 44.6 readership counts on Mendeley and are cited 10.6 times in Scopus and 8.6 times in WoS. Publications from the fields Law, arts, & humanities (13.1) and Language, information & communication (23.1) also exhibit a substantially higher density of readership in contrast to their citation densities (in both Scopus and WoS). On the other hand, Engineering science is the field with the lowest readership density (11.7), although its readership density is still higher than their citation density (8.1 in WoS and 8.2 in Scopus). These results hint to the added value of readership indicators for reflecting the impact of the fields which are typically not very well represented by citation metrics.

Another important weakness in citation analysis is the need of waiting for longer periods for publications to achieve a substantial number of citations, which challenges the citation analysis of very recent publications. Although Mendeley has been observed to be a metric with a relatively slow pace (Zahedi, Costas, and Wouters, 2017), it has been reported that Mendeley readership tend to have a faster accumulation for recent publications than citations (Thelwall and Sud, 2016). This suggests that readership could play a relevant role in informing the early impact of publications (Maflahi and Thelwall, 2016). In this section this point is illustrated (Table 4 and Figure 2; see also Supplementary file 3. Table A3 in the appendix26 for the same analysis based on the whole dataset) by studying the temporal trend in the impact of WoS publications with at least one citation in Scopus/WoS or at least one reader in Mendeley from 2012-2018.

The coverage of Crossref DOIs with at least one Mendeley readership has a steady pattern from 2012 to 2017 with a small decrease (95%) in 2018. In contrast, the coverage of Crossref DOIs with at least one citation in both Scopus (from 94% in 2012 to 62% in 2018) and WoS (from 91% in 2012 to 73% in 2018) shows a decreasing pattern over time (2012-2018). In terms of average impact, Figure 3 shows overall decreasing pattern for MRS with a little increase from 2012 to 2013 and decreases from the year 2015 onwards while MCS decreases steadily over 2012 to 2018. However, MRS is indeed higher than MCS for the whole period of 2012-2018. This result is in line with those from previous studies (Maflahi and Thelwall, 2016; Thelwall and Sud, 2016; Zahedi, Costas, and Wouters, 2017; Thelwall, 2018) and suggests that Mendeley readership can work as an important source to reflect evidence of early impact of publications.

In contrast to traditional citation indicators which do not provide much information about the citers (e.g. if citations are from PhDs, Professors, etc.), Mendeley readership data includes information on the Mendeley user types, as indicated by the users themselves in their Mendeley profiles. This information provides the opportunity to identify and characterize users and potentially distinguish scholarly and non-scholarly ones.

For this purpose, Mendeley users were grouped into seven broad user types:27 PhD students, bachelor and master students, researchers (academic and non-academic institutions), professors & lecturers, librarians, other professionals, and unspecified users. The dataset of Crossref DOI with at least one reader (n = 66,536) selected for this analysis. Coverage of publication with at least one reader and Mean Readership Scores (MRS) per user type were calculated.

Table 5 shows coverage of publications and their average readership scores by different user types. The results show that the highest coverage of publications saved by students (PhD, bachelor, and master) than other users. Also, higher MRS for the sets of publications saved by Students (PhD, bachelor, and master), over those saved by researchers, professors and other user types is visible. The same pattern is observable across the six sample universities (see Supplementary file 4. Table A4 in the appendix28). The possibility to track the use of scientific publications by different user type is an advantage that citation indicators do not have.

Figure 4 shows the average readership scores classified into different categories of impact based on the user types across the six sample universities. For this, average readership scores by aggregating similar user types into related categories has been calculated as: MRS Scientific (MRS by professors, researchers, and PhD & postgraduate students), MRS Educational (MRS by bachelor and master students), MRS Professional (MRS by librarians and other professionals), and MRS unspecified (MRS by unknown users). Based on this figure different types of impact of publication could be identified which provide useful insights in the various types of impact of publications and its trend across the selected universities.

Based on these results and those from previous studies (Bornmann and Haunschild, 2015; Mohammadi, et al., 2015; Thelwall, 2017a; Zahedi and van Eck, 2018), the identification of patterns of readership per user type could help in informing different types of impact (e.g., scientific, educational, or professional) of publications beyond the more academic types of impact reflected by citation indicators.

However, there are some limitations that need to be considered when using Mendeley users as a proxy for different types of impact. Firstly, Mendeley users are self-reported, this means that they choose their user type from a list of predefined options which may not always correspond to their local attribution (e.g. in some countries professors and researchers may be equivalent, like the CSIC in Spain and the universities in the country), and lecturers may have a different academic consideration depending on the country (e.g. in UK a lecturer can be equivalent to an assistant professor, or to a research associate in the US29). Secondly, users may change their status (e.g. a PhD may become a Postdoc or a researcher) without updating their Mendeley profile. Thirdly, the different users are not completely independent from each other, and what Bachelor & Master students read may be influenced by what is recommended by their lecturers, and PhD may read what is recommended by their senior colleagues (Postdocs and professors).30

Mendeley readership can represent an important source of evidence of the impact of publications not indexed in mainstream citation databases (WoS or Scopus). This is particularly relevant for publications from the Global South and developing countries, since they usually have a lower coverage in most bibliometric databases (Alperin, 2013), with a substantial share of the output from these countries underrepresented in citation databases (Zahedi, & Costas, 2017). In a recent study it has been observed that publications from the Global South countries (Africa and South America) have relatively higher levels of readership in contrast to citation metrics (Costas, Zahedi, and Alperin, 2019), which reinforces the idea that Mendeley readership can play a relevant role in providing impact evidence for non-indexed publications. In this report the higher coverage of University of Sao Paulo (USP) publications on Mendeley reinforces this relevance of Mendeley readership for studying the impact of Global-South countries and non-indexed publications.

A higher density of readership (over citation metrics) is typically observed for social sciences and humanities publications. In citation databases, citation impact in these disciplines is largely affected by the lower coverage of books and by the more national or local orientation of the published research (typically in other languages than English) (Moed, 2006). This study illustrates how Mendeley readership can be a valuable source to inform the impact analysis of the fields that are not well represented with citation indicators.

Reviews and articles are the most prevalent document types in citation analysis, since these document types capture the most important scientific findings. Usually other document types (e.g. editorial material, letters, meeting abstracts, news items, data papers, etc.) are excluded from citation analysis because they are deemed not to represent the same type of scientific contribution than articles and reviews. However, there may be situations in which the impact analysis of these other document types is necessary (e.g. a journal that wants to analyze the impact of its editorials or news items; or research teams that also want to evaluate the impact of those types of outputs). In such cases, citations are not very helpful given the low citedness of these document types. However, we have illustrated how Mendeley readership have a higher coverage and higher readership values for some of these document types (e.g., letters, data papers or editorial materials), supporting the idea of a strong relevance of Mendeley readership for the evaluation of these outputs.

We have illustrated how readership scores are more prevalent than citations in recent publications and hence they could work as an early indicators of research impact (Thelwall and Sud, 2016; Thelwall, 2018). These findings together with the fact that Mendeley readership are available openly and that they can earlier signal highly cited publications (Zahedi, Costas, and Wouters, 2016), highlight the value of Mendeley as a tool for revealing early impact of publications, particularly when substantial number of citations haven’t yet been accrued.

We argue and illustrate how readership scores from non-academic users (such as students, librarians, or professionals) could reflect other types of impact, such as educational or professional. This more fine-grained possibility of studying the different types of users interacting with the publications on Mendeley, is something that is not possible with the most common citation indicators. This suggests the potential of Mendeley as a relevant source for expanding the notion of impact beyond the more academic impact captured by citations, although self-reported nature of Mendeley needs to be considered.