For many scholars, the biggest disadvantage of Web of Science and Scopus is simply that neither resource is available to them. Although faculty at the major research universities often have access to at least one of these databases, the situation is very different elsewhere. Apart from those institutions in the Carnegie R1 and R2 categories, just 25% of American four-year colleges and universities provide access to either Scopus or Web of Science.2 In contrast, 65% have current subscriptions to SocINDEX or Sociological Abstracts,3 and the most popular disciplinary databases such as ABI/INFORM, EconLit, ERIC, MEDLINE, and PsycINFO are likely to be even more widely available.

Our experience at several universities suggests that for institutions with 2,000 to 10,000 students, Scopus costs three to four times as much as SocINDEX or Sociological Abstracts. Moreover, Web of Science generally costs more than Scopus. Cost is not the only factor that influences library holdings, of course. The disciplinary databases each have a clear constituency—an academic department or school with a strong interest in maintaining access to specific databases and journals. Although each multidisciplinary database may be of some interest to a large number of faculty, no single group is likely to feel a compelling need to choose Scopus (for instance) over Biological Abstracts or MathSciNet. When a small group with well-defined interests and a larger group with more diffuse interests compete, the small group is likely to prevail (Elhauge 1991; Olson 1971).

Finally, there is a perception among some librarians and faculty that Scopus and Web of Science are not especially attractive to undergraduates—that the advantages of these databases (such as size, multidisciplinary scope, and citation-tracing capabilities) are offset by disadvantages such as complicated interfaces and marketing strategies that target expert users rather than undergraduates. Faculty may appreciate the breadth of Web of Science and Scopus, but students often adopt a more constrained approach to database selection. As a student at Manhattan College stated during web site usability testing, ‘If my paper is for a psychology class, I look for a database with “psych” in the name.’

Because Scopus and Web of Science are held by relatively few U.S. colleges and universities, many researchers who use bibliographic or bibliometric data must look elsewhere.4 One might argue that this is not a major problem—that bibliometric research is more likely to be conducted by research-university faculty than by those at other institutions. However, this raises a question: Can the dominance of the R1 universities within fields such as information science be attributed to the long-term centrality of expensive databases such as Web of Science? Faculty at bachelor’s colleges may avoid bibliometric research simply because they lack access to the tools that are closely associated with it. The situation has at least two potentially negative consequences. First, the perspectives of scholars at undergraduate colleges and universities may not be fully represented within the literature. Second, undergraduate science faculty may miss an opportunity to undertake empirical research that is less expensive than most lab research, that does not normally require external funding, and that lends itself to multidisciplinary faculty-student collaboration.

A second limitation of the large citation databases is their relatively poor coverage of books and conference proceedings. Scopus, Web of Science, and Google Scholar are all devoted primarily to journal articles. For instance, Scopus covers more than 39,000 journals but just 1,628 book series, 514 conference proceedings, and no books issued independently (i.e., not as part of a series). Likewise, the three most readily available Web of Science databases—SCI, SSCI, and Arts & Humanities Citation Index—include about 1/20 as many books and chapters as journal articles. The Web of Science book and proceedings databases are much smaller, offered as separate products, and not widely held, even by research universities. Because Google Scholar makes no distinction between articles, conference papers, books, and other online resources that ‘look scholarly’ to its web crawling mechanisms, the methods used in its construction do improve its coverage of conference papers and other non-journal documents. Google Scholar’s coverage of books and chapters is still limited, however, perhaps because books are far less likely than journals to be indexed and made available online (Harzing 2019; Martín-Martín et al. 2018).

The importance of conference proceedings is well established, especially in rapidly changing fields such as computer science (Bar-Ilan 2010; Larsen & von Ins 2010; Lisée et al. 2008). Likewise, books remain central to many areas of inquiry in the social sciences and humanities (Engels et al. 2018; Giménez Toledo 2020; Giménez Toledo et al. 2013; Moksony et al. 2014; Nederhof et al. 1989). It is also important to realize that where particular social science disciplines are concerned, the exclusion of books can lead not just to the underestimation of scholarly productivity and impact, but to systematic bias on the basis of research area, institution, and institution type. Recent data for U.S. sociology faculty reveal that nearly all research-active authors can be readily categorized as article authors or book authors; the correlation between article and book counts during the 2013-2017 period is just 0.13. Likewise, nearly all academic departments can be categorized as article departments or book departments; the department-level correlation between articles per faculty member and books per faculty member is just 0.23 (Wilder & Walters 2020b, in press). These distinctions within sociology, based at least partly on subfield (e.g., demography vs. critical theory), parallel the broader distinction between the article-based sciences and the book-based humanities.

Although faculty at the major research universities have far higher average article counts than those at other institutions, the highest average book counts can be found among faculty at the top liberal arts colleges (Wilder & Walters 2020b, in press). This suggests that liberal arts faculty may have a distinctive role in consolidating, synthesizing, and popularizing sociological research. It also demonstrates the advantages of evaluating books and articles separately—something that is not possible if Web of Science, Scopus, or Google Scholar are used as the sole information sources.

A third limitation of the large citation databases is the difficulty of distinguishing among authors with similar names. To some extent, the problem can be traced to bibliographic errors in the databases themselves. For instance, Web of Science is not always consistent or reliable in its reporting of author names and institutional affiliations (Walters & Wilder 2016). The address field, which includes the author’s institution and department, is sometimes incomplete or difficult to interpret, and the information appears to be compiled just from the first page of each article, so relevant information presented elsewhere (in a biographical statement at the end of the article, for instance) is omitted. Our informal investigations suggest that authors’ first names, rather than initials, are not provided consistently for papers indexed prior to 2008, and Harzing (2013) has reported that the document type field does not always accurately distinguish between peer-reviewed articles and other contributions. Similar problems can be seen with Scopus, and the bibliographic errors associated with Google Scholar have been well documented (Delgado López-Cózar et al. 2019; Jacsó 2005, 2008, 2010; Orduña-Malea et al. 2017).5

Although the difficulty of resolving authors’ names can be addressed through author identifier systems such as ORCID and ResearcherID, many authors are not included in either system. In compiling author information for our data set, we looked for databases that (a) provide complete and accurate author information, (b) maintain their own author identifier systems (although there are no such databases for sociology), (c) cover a limited range of subject areas and therefore minimize the number of instances in which ‘unwanted’ authors appear in the search results, and (d) provide, for each record, the full text of the title page and any other pages on which bibliographic or author information is likely to appear. We also conducted each search manually rather than relying on automated procedures. This last point is discussed more fully in section 4.3.

Accurate name disambiguation is especially important for productivity studies—those in which the investigator starts with a well-defined list of authors and counts all their publications or citations, wherever they may have appeared. When conducted manually, the compilation of data for a productivity study usually involves a large number of author searches, and the correct identification of individuals is central to the process. In contrast, contribution studies are those in which the investigator starts with a well-defined list of publications and records the contributions to that literature by all authors, whomever they might be. As might be expected, the distinction between productivity studies and contribution studies has implications for the kinds of research questions that can be addressed (Wilder & Walters 2019).

When evaluating the publishing productivity of American sociologists, we found no database that provided adequate coverage of both journal articles and books. (Other scholarly works, such as conference papers, were not included in our analysis.) We chose SocINDEX as our primary source of journal article data after evaluating Google Scholar, Scopus, Web of Science, SocINDEX, and Sociological Abstracts on the basis of five criteria:

Despite their very broad coverage, none of the three large citation databases (Google Scholar, Scopus, or Web of Science) are as comprehensive as SocINDEX and Sociological Abstracts with regard to sociology (criteria 1 and 2). For instance, the Scopus sociology and political science category includes 1,269 journals, about the same number that are indexed cover-to-cover in SocINDEX (1,257). SocINDEX also provides partial coverage of more than 1,500 other journals, however, teasing out individual articles of sociological interest from journals such as Crime and Delinquency, Ethnicity, and the Journal of Biosocial Science. Finally, SocINDEX provides better coverage of the sociology journals that aim to influence teaching and practice rather than academic research—the same journals that may be especially receptive to the work of authors at bachelor’s and master’s institutions. It is also likely to include more journals of local or regional interest—those with regional influence disproportionate to their overall citation impact (Etxebarria & Gomez-Uranga 2010).

Google Scholar, with its idiosyncratic presentation of results, did not satisfy criterion 3. None of the three large citation databases satisfied criterion 5.

In many academic disciplines, a single database such as EconLit or PsycINFO is widely accepted as the foremost source of bibliographic information. In contrast, sociology has two contenders: SocINDEX and Sociological Abstracts. Although the two are comparable in many ways, SocINDEX has a broader subject scope and provides more thorough coverage of peer-reviewed journals (Tyler et al. 2017). Each indexes a similar number of articles each year (52,000 for SocINDEX vs. 51,000 for Sociological Abstracts), but SocINDEX covers more journals in their entirety (1,257 vs. 922). Our impression, based on extensive experience with both databases, is that SocINDEX provides especially good coverage of the social science fields to which both sociologists and other scholars contribute (e.g., criminology, demography, gender studies, gerontology, organizational behavior, social psychology, and social work). However, Sociological Abstracts seems to offer more comprehensive coverage of sociological topics that border on the natural sciences or the humanities, in fields such as area studies, environmental studies, geography, history, law, and philosophy.

Each discipline has its own subject databases with unique characteristics, of course. We recommend that investigators seeking the most appropriate data sources consider the literature of the relevant disciplines, the scope/coverage information available at publishers’ web sites, and the database reviews and comparative reports that have appeared in the library and information science (LIS) literature. LISTA, perhaps the most prominent LIS journal database, is freely available online (EBSCO 2021).

We evaluated nine source of bibliographic data for books. Six were withdrawn from consideration early in the process due to obvious gaps in coverage. Specifically,

Three data sources are more comprehensive, however.

Of these nine information sources, we chose Amazon.com for our study of sociologists’ publishing productivity. Although Amazon, WorldCat, and GOBI each provide good coverage of scholarly books, Amazon has a more user-friendly interface and includes title page images, which provide for reliable verification of bibliographic information. A more serious problem with WorldCat and GOBI is that they often present multiple records for a single title, making it difficult to distinguish between new (original) works and revised editions. (As noted in the Appendix, section A.7, our goal was to include new books but to exclude new editions, translations, and reprints.) With WorldCat, in particular, a single title may be represented by dozens or even hundreds of records that represent related works or that reflect the application of different libraries’ cataloging practices to works that are identical in every respect. A difficulty we recently encountered, while not typical, reveals the extent of the problem. To estimate the number of libraries with access to the Web of Science database, we initially attempted to count the number of holding libraries listed in WorldCat.6 A WorldCat title phrase search for Web of Science returned 192 records, including 109 that correspond to the entire database or to one of its primary components (SCI, SSCI, Arts & Humanities Citation Index, or JCR). Some of those records are virtually identical (e.g., one might use an ampersand rather than ‘and’ in a key field), some have been modified to comply with the cataloging standards of particular institutions or regions, and others were clearly created in error but never removed from the database. A significant number of the near-duplicate records appear to reflect disagreements among catalogers about best practices, and quite a few are unique due to local holdings information that should not have been added to the bibliographic record itself.

Although Amazon.com was the single best source of book data, we relied on additional sources—university web sites, publishers’ web sites, personal web sites, Google Scholar, and OCLC WorldCat—to verify and clarify the information for about 20% of the faculty on our list. Many sources included helpful information, but none were comprehensive and many omitted the authors’ most recent publications. In particular, authors’ online CVs were often incomplete, out of date, or potentially misleading. For instance, many listed edited volumes as if they were single-authored books.

As noted in section 1, our data file includes five-year publication counts (2013-2017) for 2,132 professors and associate professors in 426 U.S. departments of sociology. Publication counts and related data are presented separately for individuals and for academic departments. The data file, in .xlsx format, is freely available through both Zenodo and openICPSR (Wilder & Walters 2020a).7

The data compilation procedures, described fully in the Appendix, result in a data set with several unique advantages:

Our data do have five significant limitations, however. First, assistant professors are not included. Section A.5 of the Appendix presents the rationale for this decision. Second, the data for institutions in the R1 and master’s categories are sample data rather than population data. Complete population data are provided for the institutions in the other four categories, however; see the Appendix, sections A.1 and A.3. Third, our reliance on a list of authors working in departments of sociology, and on SocINDEX, limits the extent to which our data can be used to evaluate interdisciplinary topics. Databases such as Scopus and Web of Science continue to have a clear advantage in that respect. Fourth, our methods cannot capture the most recent publications in a reliable way. Some books appear in Amazon months before their publication date while others appear only afterward, and this uneven coverage creates the potential for bias unless the publication cut-off date is set at least a few months before the start of the data compilation process.9 Finally, while our data include the journal name, journal CiteScore, and year of every article, article titles and DOIs are not included. This makes it difficult to link each individual article to the article data available elsewhere—to article-specific citation counts, for instance. For books, the title, publisher, and year are provided.

Overall, six types of variables are included in the data file: general data on academic institutions (sociology departments), general data on individuals (sociology faculty), data on each journal article, data on each book, productivity data for individuals, and productivity data for academic institutions. For a list of the variables, see the Appendix, section A.9.

Although at least 25 studies have rated or ranked the scholarly output of sociologists and sociology departments since 1970, just three post-2000 analyses include rankings of at least 40 U.S. sociology departments based on articles published in a wide range of journals (Wilder & Walters 2019, Table 1). Of the three, two deal exclusively with research universities (Ostriker et al. 2011; ShanghaiRanking Consultancy 2021) and one deals exclusively with liberal arts colleges (Hartley & Robinson 2001). None account for publications other than journal articles. Our data set therefore provides for a broader approach to the assessment of particular authors and departments. However, its real value lies in the inclusion of key covariates and the provision of identifying information that can be used to link these data to information obtained elsewhere. That is, our data set can be used to study the relationships between individual characteristics, institutional characteristics, and publishing productivity.10 For instance, the first study to use these data (Wilder & Walters 2020b) presented several new findings:

The second study based on these data focused on the most productive faculty at various types of colleges and universities, yielding further evidence in support of the second finding, above (Wilder & Walter, in press). It also revealed that while the most productive authors, as a group, tend to publish in the same journals as other faculty, they are especially likely to publish repeatedly in their own preferred journals, which vary with each individual.

We encourage others to use our data set, either to explore new areas of research or to investigate these same topics in greater detail or from different perspectives. As noted in section 4.1, these data may be of limited value for multidisciplinary research. At the same time, however, our data—and our methods, more generally—are well suited to research on academic or professional groups that can be clearly delineated on the basis of individuals’ characteristics or the characteristics of their publication outlets. Although bibliometric research often deals with interdisciplinary or multidisciplinary groups, an approach centered on particular disciplines or occupations may be more appropriate for research in fields such as labor economics and the sociology of professions.

The methods used to compile and clean the data are described in the Appendix. To summarize, we began by identifying the sociology departments in the population of interest—four-year public and nonprofit colleges and universities in the United States—and compiling rosters of all the faculty with professor or associate professor rank. We then searched SocINDEX, Amazon, and other publicly accessible sources (e.g., course catalogs, Google Scholar, the IPEDS Data System, OCLC WorldCat, personal web sites, ProQuest Dissertation Express, publishers’ web sites, and Scopus Sources) to compile information on institutional characteristics, individuals’ characteristics, and publishing productivity.

These methods are time-intensive, of course. Although we did not systematically record the time spent on each particular task (i.e., the average time per individual for the faculty rosters or the average time per article for the journal article data), we can provide a general sense of the time commitment required. Working 15-20 person-hours per week on data compilation, we spent 11 weeks compiling departmental rosters for 426 departments (2,132 individuals), 14 weeks compiling data on 4,928 journal articles, and 10 weeks compiling data on 598 books. That’s roughly 5 minutes per person record, 3 minutes per article record, and 18 minutes per book record.11 In general, information about faculty at the major research universities could be obtained far more readily than information about faculty elsewhere. Moreover, a relatively small number of individuals—perhaps 15%—accounted for much of the total time spent investigating and verifying author and publication information.

Many information science researchers use automated methods to compile bibliographic and bibliometric data. For instance, application programming interfaces (APIs) can be used to harvest data from Google Scholar, Scopus, and WorldCat (Elsevier 2021; OCLC 2021; SerpAPI 2021), and some data sources are specifically designed for use with APIs (Hendricks et al. 2020; Visser et al. 2021). There are four problems with the use of APIs and other automated methods, however. The first is that automated searching relies heavily on the accuracy of the data, especially the subject codes. For instance, if the subject codes are defined poorly or applied inconsistently, they will not extract the records that the researcher desires. As discussed elsewhere (Walters 2017), the large citation databases sometimes group two or more distinct fields of study under a single code. Journals from a single discipline are sometimes split across multiple subject categories, some subject categories are much narrower than others, and some do not seem to correspond to coherent research areas or disciplines. Moreover, the large citation databases are simply prone to error. Scopus once classified Developmental Psychology as a demography journal, for instance, and Web of Science once listed Financial Research Letters in the infectious diseases category (Jacsó 2011, 2012).

A second problem is the existence of multiple records for a single research contribution. In some cases, the papers are variants such as a published article and the corresponding manuscript or preprint, but the database provides no mechanism by which the researcher can choose to count these variants as the same work or as multiple works. In other cases, the exact same paper (e.g., the same PDF file) is represented by multiple records. As discussed in section 3.2, this is a particular problem with WorldCat, which has 109 records for Web of Science and its key components. Disentangling the relationships among these records requires considerable effort, and it cannot be done through an automated search mechanism. The same problem is readily apparent with Google Scholar.

A third difficulty is that many automated methods fail to retrieve all relevant records, often due to a lack of standardization in the underlying data. For an investigation of the citation impact of papers in predatory accounting journals (in progress), we evaluated the performance of the Publish or Perish search tool (Harzing 2016) in retrieving Google Scholar records for articles published in the International Journal of Accounting and Financial Reporting from 2015 through 2018. The Journal’s web site lists 209 papers within that date range, and manual searches of Google Scholar for each article title retrieved 193 of them. However, a July 2020 Publish or Perish journal title search retrieved just 108. Further investigation revealed at least five reasons for the discrepancy: (a) many journal titles are very similar, and some of them fully incorporate the titles of other journals; (b) many Google Scholar records use abbreviated journal titles, often with several different abbreviations for a single journal; (c) journal titles are not always consistent, even at the publishers’ web sites;12 (d) automated journal title searches exclude variants of the article that are not labeled with the journal name, such as pre-acceptance manuscripts and working papers; and (e) some individual articles are simply not included in Google Scholar.

Finally, automated searches may lead researchers to bypass the mechanisms that would otherwise make them more familiar with the patterns and idiosyncrasies that exist within the data. We believe manual searching can give the investigator a deeper understanding of the relationships among variables, including facets of those relationships that might not be detected by the more common statistical methods. (For example, is there a threshold level at which institutional prestige begins to influence book productivity? Is the relationship between gender and article productivity conditional on a third characteristic? Do the a priori delineations of the variables capture the most important distinctions between institutions and groups, or would alternative specifications be more appropriate?) Likewise, the experience of compiling the data may suggest hypotheses, explanations for findings that emerge later in the research process, or caveats related to data interpretation that might not have come to light through automated searching. For example, when compiling the data we noticed that many mid-ranked bachelor’s and master’s universities have just one ‘superstar’ faculty member with far higher productivity than the others—and that a disproportionate number of those superstars are women. We interpreted our initial statistical results with this idea in mind, then later developed more careful, formal methods of evaluating the situation (Wilder & Walters 2020b, in press). We cannot claim that manual data compilation is always a cost-effective approach, but it does appear useful as a mechanism for stimulating the kind of associative thinking that can lead to new insights and perspectives (Benedek et al. 2012; DeHaan 2011; Mednick 1962; Verhaeghen et al. 2017).