Performance evaluation plays an important role in management. Performance evaluation is presented as a process of quantification of the performance and effectiveness of actions (Hansen 2017). It plays a key role in building institutional strategies, funding schemes, recruitment and career promotion issues.

Research management is quite a creative process. There are no exact standards that allow an objective and precise assessment of research performance. The assessment of individual researchers remains a deliberately difficult process without a standard solution.

Research performance evaluation has a range of goals. There are numerous studies on the impact of research performance evaluation. Some studies state that employees benefit from performance evaluation which leads to greater motivation (Terlicka 2013, Tuytens & Devos 2012). However, others conclude that without clear goals the performance appraisal can cause employee dissatisfaction and even a reduction in organisational commitment (Maley 2013). Fair performance assessment has a significant impact on organizational excellence and competitiveness in the long run (Arbab & Abaker 2018). When considering different views the link between performance evaluation and employee motivation tends to become blurred (Idowu 2017).

Nowadays, in research performance evaluation the two main approaches used are peer review and bibliometrics. The latter tends to be more popular due to a growing volume of open information published on the internet. The Web of Science and Scopus citation databases are the main tools available for bibliometric assessment. Bibliometric measures derived from these databases became target indicators for the performance assessment of researchers, organizations and countries. Global ranking agencies explore the opportunities of bibliometrics which are used to serve as evaluation criteria. One of the criteria of the Global Competitiveness Index is the number of publications and their citations in the Scopus database, while the Global Innovation Index is based on citation indices derived from the Web of Science. The world university rankings (for instance, QS, Times Higher Education, ARWU) also push the research institutions and universities to publish in sources indexed in Web of Science or Scopus. Bibliometrics has become entrenched in the individual, team and institutional assessment of research performance.

Obviously, inappropriate indicators create perverse incentives (Wilsdon et al. 2015). Excessive bibliometric pressure increases publications in the predatory journals.1 In the case of Australia, the bibliometric approach of research performance evaluation policy showed a drop in the relative quality of publications along with an increase in publication productivity (Butler 2003). In Turkey it led to uncertainty, and to research ethics and quality issues (Yucel & Demir, 2018). Similarly, Kazakhstan appeared in the list of the most affected countries (Macháček & Srholec, 2021). Such misuse of bibliometrics led to the publication of the Leiden Manifesto which comprises ten principles that guide best practice in metrics-based research assessment (Hicks et al., 2015). Unfortunately, most governments’ policies do not comply with the Leiden manifesto principles and bibliometric targets continue to prevail and lead to low-quality publications and research ethics violations.

Moreover, researchers do not only carry out publishing, they engage in a lot of other activities: reviewing, organizing conferences, writing proposals, supervising students research, etc. Despite their high significance, these kinds of activities are not taken into account when measuring research performance. While promoting “inclusive metrics of success and impact to dismantle a discriminatory reward system in science”, Davies et al. (2021) distinguish between a narrow and an inclusive view of scientific impact. The narrow view is focused on citation in evaluating scientific impact, while the inclusive view captures all activities: education, collaboration, science communication, community engagement, etc.

The current research adopts the inclusive view: the main issue is to recommend inclusive performance assessment criteria through assessing the perceived significance of each type of activity. The inclusive performance assessment motivates high quality of research, high level of productivity and strong performance in all areas.

Therefore, this research aims to assess the importance of different types of activities and metrics from the point of view of researchers in order to recommend an inclusive set of research performance criteria.

The two main approaches of research performance evaluation are peer review and bibliometrics. The debates over the pros and cons of both approaches have continued for about three decades. Here we discuss the recent research findings on both methods.

Peer review is a qualitative tool. The main drawbacks of peer review that have been observed over time are subjectivity and expensiveness. Similarly, the current peer review system has been criticized for failures in representing the opinions and interests of non-peer clients (Frijters & Torgler 2019). Researchers still find biases in peer review processes (Santos et al., 2021; Haffar et al., 2019). Some studies confirm the contribution of the peer review system to gender inequality (Sato et al., 2021). Tamblyn et al. (2018), for instance, showed that female applicants were found to receive consistently lower reviewer scores than male applicants.

Despite the inconsistency of the h-index and the inappropriateness of the journal impact factor for research performance assessment (Waltman & van Eck, 2012; van Leeuwen, 2012), the bibliometric approach to research assessment has become popular. It is most frequently performed in the Nordic countries, the Netherlands, Italy and the United Kingdom (Jappe, 2020). Bibliometrics are considered more attractive than peer review because they demand much less time and effort. Moreover, some studies confirm that they are consistent with peer review results. Thus, policymakers increasingly use these metrics in addition to, and in some cases instead of, expert assessment of research results (Belter, 2015).

However, the stand-alone use of bibliometrics brings distorted results. The emphasis on publications in high impact factor journals diverts the attention of researchers from the real contribution to knowledge production. They shift their focus to tactical actions to match established reward and promotion systems (Mahmood & Shah 2016). Evaluation using bibliometric indicators can be also complicated and unreliable due to the high probability of misuse. The interpretation of these indicators should be done by field experts. Otherwise, the use of metrics can lead to the demotivation of researchers and a decrease in the quality of research and even education. This is because unlike scientific productivity, the quality of research does directly affect the quality of education (Cadez et al., 2017). Also, bibliometric indicators can be manipulated, causing problems in measuring societal impact (Moed, 2017). Some authors claim that “citations and impact factors metrics are flawed and biased against already marginalized groups and fail to accurately capture the breadth of individuals’ meaningful scientific impacts” (Davies et al., 2021). Moreover, the “quick-and-dirty” use of metrics can introduce inaccurate results into the assessment of academic research performance (Moed, 2020).

The dominance of the bibliometric approach in research performance evaluation has both short and long-term consequences such as encouraging plagiarism, “salami slicing”, demotivation of young researchers to write papers due to slim chances of publication in high-impact-factor journals, etc. (Lawrence, 2003; Cherubini, 2008; Notkins, 2008). Among the long-term consequences Brischoux & Cook (2009) mentioned a reduction of the amount of information available to the scientific community because of the failure to publish in high-impact-factor journals and the suffering of aspects of research such as core creativity if scientists only focus on their publication records.

To apply bibliometric methods it is essential to ensure that international journals are a major means of communication in each particular field (van Raan 2014). If the major scientific communication is through national journals, the bibliometric approach is not applicable. In the bibliometric approach, one should take into account that industry research has a lower impact than public research (Abramo, 2021). Hence, it must be evaluated in a different way than public research. Evaluators also should take into account that correlations between technological and scientific impact indicators are mostly low or moderate (Guerrero-Bote, Moed & De-Moya-Anegón, 2021). This shows that they measure different aspects of research performance. As the use of the h-index alone causes a loss of citation information, Zhang proposes the introduction of the e-index (Zhang 2009). When applying the bibliometric approach the phenomenon of “delayed recognition”, also known as the “Sleeping Beauty phenomenon”, should be taken into account. “Sleeping Beauties” are publications which may attract a lot of attention and become cited after many years (van Raan, 2004). It has been noted that publication in open access journals positively influence citation impact (Moed, 2007). Therefore, bibliometrics cannot indicate anything if the scientific field is not represented sufficiently well in open information space or if information policy prohibits posting publications in open access directories. For bibliometric analysis in the social sciences and humanities, a broader range of both publications (including non-ISI journals and monographs) and citation indicators (including non-ISI reference citation values) is needed (Nederhof, 2006).

Based on the literature review, it should be noted that the methodology of bibliometric assessment has a number of advantages over the method of expert assessment: it is inexpensive, non-invasive, easy to implement, provides both rapid updates and intertemporal comparisons, is based on objective quantitative and qualitative data, and provides a high degree of representativeness. Citation metrics can make the process more efficient and cost-effective, but expert evaluation should remain a central element in any research evaluation process (Butler, 2007). The challenge is to combine these two methodologies so that they work effectively together (Moed, 2007). Peer review supported by advanced bibliometric methods can make the process more objective and transparent (van Raan, 2014). To avoid the subjectivity of peer review-based evaluation ‘disciplined peer review’ has been suggested (Besselaar & Sandström, 2020).

One instance where the bibliometric approach can replace peer review is in assessing groups (Ball, 2020). This is possible due to bibliometric methods based on advanced application tools, which have become high quality, reliable and very informative instruments in research performance evaluation. However, evaluating individual research performance is a complex task that would ideally examine productivity, scientific impact, and research quality, a task that metrics alone have been unable to fulfill (Sahel, 2011).

The literature review on approaches to research performance assessment shows that both the bibliometric and expert review approaches have their advantages and disadvantages. Recent studies show that there is a trend to combine these two approaches to achieve synergy in research performance evaluation results.

The science evaluation policy reforms in Kazakhstan started in 2011 when the country joined the Bologna education system and announced new PhD thesis requirements of publication in WoS or Scopus indexed journals. That led to the boosting of publications in the predatory journals.

Later, Kazakhstan continued, step by step, its policy of introducing bibliometrics into research project evaluation, settling requirements for research principles, evaluating candidates for the conferring of academic titles, etc. As a result of the policy, Kazakhstan has entered the list of the countries most affected by a reliance on bibliometrics (Macháček and Srholec, 2021).

Table 1 presents the number of publications per year made by authors from institutions located in Kazakhstan during 2000-2020. The table shows that the number of publications increased rapidly in 2015 compared to 2010. This increase took place in 2013. While during 2000-2012 Kazakhstan’s rank in the Scimago list was in the range 90-100, as from 2013 it increased to 70, and in 2014 to 66, remaining more or less stable up until 2020.

When tracking changes to the national policy of research performance evaluation for different purposes (conferring PhD degree, academic titles, granting funds for projects, etc) we can confirm that the bibliometric approach has become more sophisticated year on year.

The policy related to conferring PhD degrees is still changing and its history resembles the overall policy trends in research performance evaluation. At the initial stage the policymakers put general publication requirements such as publication in Web of Knowledge journals with a non-zero impact factor or indexed in Scopus (Table 2). In the second stage (in 2016), they started to differ between research fields. The third stage of amendments differentiates between ranks of journals (quartiles and percentiles) in which PhD candidates have to publish.

Table 2 demonstrates similar changes in the Expert Council membership requirements. In 2018 a requirement on having publications and citations in Web of Science or Scopus was introduced. Another option is having publications in local journals approved by the national authority. This confirms that the local authorities realize that not all candidates to the Expert Council can have publications in Web of Science or Scopus but can nevertheless be considered experts. There are no such options for conferring PhD degrees.

When the requirements for principal positions in research projects are analyzed it can be seen that they differ not only between research fields but also between types of research. For basic research, the publication requirements are higher than they are for applied research.

In natural, technical, life sciences and medicine, the requirements for project principal positions are stronger than they are for other sciences. Within the last 5 years candidates must have:

Basic research: at least 2 articles and/or reviews in peer-reviewed scientific journals included in Q1-Q3 of the Web of Science and (or) have a Scopus Cite Score percentile of at least 50, of which at least one as the corresponding or the first author.

Applied research: at least 2 articles and/or reviews in peer-reviewed scientific journals included in Q1-Q4 of the Web of Science and (or) have a Scopus Cite Score percentile of at least 35, of which at least one as the corresponding or the first author.

Expectations on output of research grants in natural, engineering, life sciences and medicine also differ between types of research and are higher than they are for other sciences:

Basic research: at least 3 articles and/or reviews in peer-reviewed scientific journals of Web of Science Q1-Q3 and/or have a Scopus CiteScore percentile of at least 50; or at least 2 articles and/or reviews in peer-reviewed scientific journals included in Web of Science Q1-Q2 and (or) with a Scopus CiteScore percentile of at least 65; OR at least 1 article and/or review in a peer-reviewed scientific journal with a percentile in the Journal Citation Reports of more than 90 or a Scopus CiteScore percentile of at least 95;

Applied research: at least 2 articles and/or reviews in peer-reviewed scientific journals, included in Web of Science Q1-Q3 and (or) have a Scopus CiteScore percentile of at least 50; - OR at least 1 article and/or review in a peer-reviewed scientific journal included in Web of Science Q1-Q3 and (or) have a Scopus CiteScore percentile of at least 50, and at least 1 foreign or international patent included in the Derwent Innovation database.

In social, humanities and military sciences the expected output includes at least 1 article and/or review in a peer-reviewed scientific journal of Q1-Q3 Web of Science, and (or) included in the Social Science Citation Index or Arts and Humanities Citation Index, and (or) having a Scopus CiteScore percentile of at least 35. The lower requirements for social and humanities sciences are due to their bias in favor of English language journals. (Moed, 2007).

Thus, the research performance evaluation system in Kazakhstan tends to strengthen the bibliometric-centered approach. The recent improvements have been related to differentiations in field and types of research.

Researchers have indicated two approaches to studying and assessing research performance: evaluative and explanatory (Avital & Collopy 2008). Explanatory performance assessment has an objective of predicting performance, it is proactive and its rationale is to improve productivity.

Our study can be considered as an explanatory performance study as it has the objective to build the knowledge society by constructing an inclusive scale for performance assessment in the academic and research sector. This approach is predictive due to the embedded feature of motivating researchers to build capacity through supervising students and growing professionally.

Our research is conducted in two stages:

There are 264 people included in the sample. The response rate is 63%. The statistical analyses are performed through the IBM SPSS Statistics 22 package program. Frequency distributions (number, percentage) are given for numerical variables while descriptive statistics (mean, standard deviation, median, minimum, maximum) are given for categorical variables (eg, gender). Normality assumptions of numerical variables are examined with the coefficients of skewness and kurtosis and these coefficients are found to be within a ± 2 range. Therefore, parametric statistical methods were used in the study. The differences between two independent groups (e.g. gender) are examined by an Independent Sample t Test. The differences between more than two independent groups (e.g. age groups) were analyzed by the One Way Analysis of Variance (ANOVA). Tukey’s multiple comparison test was used to determine from which group the difference arises as a result of a one-way ANOVA. The statistical significance level was set to p < 0.05 and the analysis was completed at a 95% confidence level.

The analysis has shown the following make-up of respondents (Figure 1)

In addition, the average age of the respondents is 43.4 years, the youngest is 23 years old, and the oldest is 84 years old. 63.7% of them are female. In addition, 55.7% are from the Al-Farabi KazNU faculty and 44.3% are from various public research institutes (Institute of Economics, Institute of maths, Institute of mechanics and machine science, Institute of oriental sciences, Institute of ICT, Institute of language sciences, Institute of philosophy, political sciences and religion studies).

The activities surveyed are classified into five groups: Publications, Public Recognition, Scientific & Organizational activities, Professional Advancement, Supervising. The content of each group is seen in Appendix 1.

The overall results of the survey are given in Figure 2.

According to the survey, Supervising (4.06) is perceived as the most “important” by the respondents. One can question this outcome since 55% respondents come from the university and this cannot be true for researchers of public research institutions (PRI). According to Table 3, there is no statistically significant difference between the institutions in terms of activity scores as a result of the Independent Sample t Test (p > 0.05).

Professional Advancement (3.99) can be interpreted as “important” as well for both university faculty staff and PRI researchers. The group of Scientific & Organizational activities has the lowest value as seen in Figure 2.

One can doubt if Public Recognition can be referred to as an activity since this group is a result of an activity or an achievement rather than an activity itself. But we are studying attitudes of researchers to different criteria. Therefore awards (e.g. public recognition indicators) can be considered as criteria as well. Figure 2 shows it is perceived as less important.

Table 4 shows that there are no statistically significant differences between age groups in terms of activity scores except Public Recognition acitivity. More specifically, one way the ANOVA test reveals statistically significant differences among the age groups is in terms of Public Recognition scores (p < 0.05). For example, the Public Recognition scores for the 51 and over age group are significantly lower than those for the 23-30 and 31-40 age groups.

To check the relationships between the activities we applied correlation analysis, which demonstrated a moderate positive linear correlation between the criteria scores (Table 5).

More specifically, there is a moderate positive correlation between the scores of Scientific & Organizational (S&O) activity and other groups.

Also, a statistically significant positive correlation exists between: 1) Public Recognition group scores and Supervising and Publications group scores; 2) Supervising and Publications criteria scores (Table 5).

Table 5 demonstrates that S&O activities correlate to all other activity groups. Moreover, Professional Advancement helps to enhance S&O activities, and vice versa. Hence, the results indicate that S&O activities are the core activities which correlate with all other activities.

One more logical finding is the correlation between Publications and Supervising activities. Supervising PhD students helps their research supervisors to publish more papers. And vice versa - if the researchers have a high publication activity, this indicates their capacity, and hence they are assigned as a research supervisor to more students. By way of comparison, the Higher Education Council of Turkey’s criterion for higher education institutions is “the number of students in master’s and doctoral programs, the number of doctoral specialities, and the number of doctoral graduates”.

The correlation between Publication and Public Recognition criteria seems plausible and clear as well. According to the national rules for awarding academic degrees and ranks, one should publish a certain number of papers to be nominated for the academic rank of professor or associate professor. To apply for a PhD thesis defence, doctorates must have at least one publication in journals of a certain quartile/percentile in the Web of Science Core Collection or the Journal Citation Reports or Scopus. In the current year, 2021, a new amendment has been added to the national rules for doctorates, according to which there is an option to skip writing a PhD thesis if a PhD student has two original research papers in the 1-2 quartile journals indexed in the Journal Citation Reports.

The One Way ANOVA test reveals no statistically significant differences between age groups in the various types of activity, except in Public Recognition activity (p < 0.05). For example, Public Recognition scores for the 51 and higher age group are significantly lower than those for the 23-30 and 31-40 age groups.

An independent Sample t Test (p < 0.05) reveals no statistically significant differences between females and males in terms of activity scores except in Public Recognition activities. Public Recognition scores of females are significantly higher than those of males (Table 6).

The activities within each group are also assessed by respondents. Figure 3 demonstrates the activities which scored as “important”.

The results of the survey demonstrate that the priorities of local researchers correspond to national and international priorities (publishing papers in local and international peer-reviewed journals indexed in WoS and Scopus, monographs). But respondents gave the highest preference to participation at overseas and international seminars and workshops, speeches at overseas conferences, seminars, and round tables. It indicates a desire of researchers to disseminate their research findings internationally and to build international links. By way of comparison, the Higher Education Council of Turkey encourages the internationalization of research by applying the criteria of “joint projects”, which is estimated through the number and budget of projects with international cooperation (both ongoing and completed).

The respondents are also aware of the importance of publishing articles in journals indexed by the Web of Science or Scopus. According to the analysis the average Likert score of the activity “publishing in first and second quartile (Q1-Q2) journals” is slightly higher (4.05) than for “publishing in Q3-Q4 journals” (3.94).

The theoretical and practical analysis of various concepts and tools for assessing research performance shows the growing importance of bibliometrics. Bibliometric criteria remain popular in quantifying the impact of a scientific article (Aragón, 2013).

The performance evaluation system which is dominantly based on bibliometrics is one-sided and biased. Such a system harms science itself as it inclines a scientist to make more publications and citations instead of doing high-quality research. Quality of research should be the primary concern when practising science and the publication of its results whilst important is a secondary concern.

Thus, peer review remains the main method of evaluating the researcher although it may be biased due to subjective factors such as conflicts of interest, lack of research competence, and superficial expertise.

However, one should keep in mind that the production and dissemination of scientific knowledge should primarily provide social and economic benefits through innovations that are not directly reflected in bibliometric scores. It is also essential to take into account that “Sleeping Beauties” can suddenly be awoken many years after publication (van Raan, 2004).

Our findings revealed that “Supervising” and “Professional Advancement” activities have a higher importance among all criteria groups. However, within the “Publication” activities, publishing papers in privileged local and international peer-reviewed journals and conference proceedings indexed in WoS and Scopus, and monographs are of a high level of importance as well.

“Publication”, “Public Recognition” and “Scientific & Organizational” activities are assessed as moderately important. Unsurprisingly, the “Public Recognition” score is for older groups significantly lower it is for younger groups.

Surprisingly for the Asian continent the Public Recognition score for females is much higher than for males. One possible reason for this could be that there is an underestimation of females in conferring ranks, titles and other awards. This hypothesis can be used for future research.

The interesting and specific point is that S&O activities correlate to all the other activity groups. There is a moderate positive correlation between the scores of Scientific & Organizational (S&O) activity and other groups. We can suppose that S&O activities are the core activities which correlate to all other activities. And the role of S&O criteria is definitely underestimated in the performance assessment of researchers. This confirms the earlier conclusion about the necessity to take S&O criteria into account when evaluating researchers’ performance. It is supposed that they lead to a more broad and intensive dissemination of research findings. This issue could be a subject of future research as well.

Analysis of sub-activities within each group showed the importance of building scientific communications internationally. High importance of “Participation at foreign and international seminars and workshops” and “Speeches at foreign conferences, seminars, round tables” indicates the welcomed willing of researchers to disseminate their research findings internationally and to build international links.

The study expands the knowledge base on the academic human resource management aspects, and can be of high relevance for substantiating the performance assessment of researchers criteria by HR managers of universities and public research institutions. The research results can be helpful for complying and coinciding individual and institutional criteria for research performance evaluation. This study suggests an inclusive scale to measure the performance of researchers. The inclusive scale for research performance assessment can motivate researchers to achieve organizational goals in a less resource-consuming way. Balanced intentions of university policies and researchers can allow the enhancement of research stimulus and increase research quality.

The additional file for this article can be found as follows: