In 1830 a revolt led to the secession of the Southern Provinces from the Netherlands and the establishment of Belgium as an independent state. It unites Flanders, the Dutch-speaking northern and Wallonia, the French speaking southern region, with Brussels as its bilingual capital. In 1920 the East Cantons with a German speaking population were annexed by Belgium following Germany’s defeat in World War I and the subsequent Treaty of Versailles.

Belgium’s population is 11.6 million: 6.7 million are living in Flanders, 3.7 million in Wallonia including the nearly 80.000 German speaking inhabitants and 1.2 million in Brussels.

Tensions between the Dutch-speaking and French-speaking population runs like a thread through Belgium’s history. Often presented as linguistic quarrels, its root causes are cultural, social and economic. To keep these conflicts manageable the centralized state was gradually transformed into a federation with uncommon characteristics. (Algoed and van Nieuwenhove, 2011; Van den Wijngaert, 2011). Political power resides at three levels: the federal state, the three linguistic communities and the three regions. This complex setting is mainly due to the bilingual character of the Brussels-Capital Region that is included in both the Flemish and French communities.

A peculiar aspect of Belgium’s federal state structure is that not only the federal level but also the regions and the communities have treaty-making power for matters within their competences.

The division of competences between the three levels is not always clear-cut. To avoid frictions and even stalemate, consultative mechanisms are created and the final say lies with the Constitutional Court of Belgium.

The devolution of competences from the federal state is not finished, as regions and communities demand more autonomy to develop tailor-made solutions for their socio-economic needs.

Since 1988, Flanders has been responsible for all aspects of higher education, except for the setting of minimum requirements for awarding qualifications that remain a federal competence.

Today, Flanders has a binary educational system. It has five universities: three comprehensive universities (Ghent University (UGent), KU Leuven, Vrije Universiteit Brussel (VUB)), the University of Antwerp (UAntwerp), and the Hasselt University (UHasselt). The latter two offer only a limited number of educational programs. Universities deliver academic bachelor’s and master’s degrees and PhD degrees; the ius promovendi is the exclusive competence of the universities.1

There are thirteen Flemish university colleges. The university colleges organize professional bachelor’s and associate degree programs. The latter is situated between secondary education and professional bachelor’s programs (European Qualification Framework Level 5). Unlike the other academic programs, the artistic programs are not integrated with the universities but within the university colleges in separate schools of arts. Research in these schools is organized jointly with the universities.

Each university college is linked to a university. The networks of university-university colleges were set up in response to the Bologna Declaration to transform the educational programs from the existing ternary degree system into the bachelor-master structure (Huisman and Mampaey, 2017). The difference between professional and academic bachelor’s programs lies in their orientation. The former is practice-oriented and focuses on the competences necessary for specific professions, whereas the latter provides stronger theoretical foundations for further studies at a master’s level. The Flemish decree (i.e. regional law) stipulates that universities and university-colleges not only provide education and training but also carry out research and provide services to the community. The scope of the research activities follows the orientation of the educational programs.

To assure that all higher education degree programs meet basic quality standards — a necessary condition for the recognition of degrees and titles, and wherever applicable, for the public funding of programs — the Dutch and Flemish Parliament has set up the Accreditation Organization of the Netherlands and Flanders (NVAO).

In contrast to most member countries of the Organization for Economic Co-operation and Development (OECD), during the 20^th century no large public institutes for applied research were set up in Belgium to support the development of leading technologies and strengthen the industrial base. A center for nuclear research established in 1952 was the only exception. In this setting, the universities became the natural partners of industry in research and innovation activities.

Another characteristic of Belgium’s research and innovation landscape were the rather low public expenditures for research and experimental development (R&D); in the mid-1980s, 1.5% of the GDP was spent on R&D; the private sector contributed roughly 2/3 to this expenditure, whereas the public sector contributed 1/3.

Successive constitutional revisions transferred the legislative power and the public funding for education and, for the large part, science and innovation, to the regions and communities. The 1988-1989 revision was a pivotal moment, with communities receiving the lump sum funding of the universities and nearly all funding for curiosity-driven research. However, it should be mentioned that some competences remain at the federal level. Even today, the universities are still funded by the federal government to carry out research projects and to participate in initiatives it deems necessary to exercise its competences. A detailed description of the partition of the competences related to R&D between the federal level and the regions and communities is beyond the scope of this paper.

With the institutional reform of 1988-1989, Flanders received both the legal competences and the financial means to develop its own science and innovation policy. Early on, the Flemish authorities recognized the critical importance of research and innovation for the region’s economic development and its population’s well-being. At the beginning of the 1990s, a decade before the European Union (EU) in 2002 set the objective of investing 3% of its GDP in R&D, the Flemish government started to systematically increase expenditures on science and innovation. For example, between 1993 and 2019, the outlay for R&D in the Flemish budget increased six fold from 327 to 1952 million euro (EWI, 2020).

To allocate the resources, a limited number of strategic priorities were set, appropriate funding instruments and agencies were established, performance was monitored, and, if necessary, remedial measures were taken.

The federal law on the universities of 1971 stipulated that each Belgian university received a lump sum based on its weighed number of students. Although slightly modified over the years, the funding formula was still used at the end of the ’80s to calculate each university’s lump sum.

It is important to emphasize that the salaries of the tenured staff were paid out of the lump sum and that there were only a few hundred other permanent positions for researchers at Belgian universities. Besides covering the cost of teaching and the general operating cost of the universities, the lump sum also provided substantial financial support for research; project-based research funding was still in its infancy.

Due to the oil shock and subsequent lax budgetary policy, successive Belgian governments were confronted with a severe financial crisis in the 1970s and 1980s. As a direct consequence, the annual adjustments of the funding formula’s elements were very often considerably below the real increase in the salary and operating costs.

Around 1985, the federal authorities became aware that R&D was key to combat the economic and financial crisis they faced. A plan was drawn up, and one of its components was the allocation of a small additional grant to each university to fund research activities. This grant had to mediate, at least partially, the impact of the erosion of the lump sum on the universities’ research capacities (Vandersmissen et al., 2015).

In 1991, the Flemish parliament used its recently obtained legislative authority to change the universities’ funding formula. The 1991 decree on the Flemish universities implemented a dual-track approach: modernizing regulation and introducing a new funding model. It not only considerably increased the autonomy of the universities and simplified the rules and regulations but also placed more emphasis on the institutions’ public accountability. The universities were given the legal obligation to systematically review all their activities and report the outcomes to the Flemish government. To comply with this regulation, the universities decided among other things to jointly set up a system of quality management at the level of the study programs. There was no similar initiative for their research activities.

Apart from substantial simplifications, the main difference between the new funding formula and the 1971 law was the considerable reduction of the impact of the evolution of the number of students: Only 50% of the total lump sum allocated to the Flemish universities still depended on the weighted number of students. Reducing the weight of the number of students was motivated by the two-fold conviction of the policymakers and the universities’ academic authorities to hedge, at least partially, the risk of decreasing student enrollment and to improve the collaborative environment among the Flemish universities with objectives to create a disincentive for aggressive student recruitment, to stimulate the introduction of inter-university study programs, and to foster the winding-down of degree programs with very low numbers of students.

The latter remained largely wishful thinking and the first assumption turned out to be erroneous. In the two decades following the introduction of this funding model the total student population at the Flemish universities increased substantially (Table 1). This trend was not uniform over the universities, with the strongest increase being witnessed at the UGent. However, due to the funding formula, the additional weighed students were only funded at a reduced rate of about 50%, resulting in a growing disadvantage to the UGent compared to the other Flemish universities.

On top of that, although less severe than in the 1980s, the annual adjustment of the lump sum below the evolution of salary and wage costs and price inflation slowly but certainly impeded the development of the Flemish universities. As already indicated, this is a structural problem originating in the 1970s and persisting still today, notwithstanding changes in the funding formula.

During the first decade of this century, the Flemish higher education landscape was profoundly transformed. Parallel with the implementation of the Bologna Declaration (1999) and the establishment of the associations of university-university colleges, in 2000, the Flemish public authorities decided to increase the lump sum allocated to the Flemish universities by more than 30 million euros over a four-year period; a substantial part was used to reduce the perverse effects of the 1991 decree. To create financial stability during this period of major institutional and educational reform, the lump sum was decoupled from the evolving number of students. As the preparation of a new funding formula for the higher education sector turned out to be more time-consuming than initially projected, the decoupling was prolonged until 2007.

The additional funding for the total lump sum only partially corrected the imbalance between the universities and the erosion of the university funding due to the insufficient correction for the increase in the personnel and operation costs. The former had the largest impact on the UGent. Claiming unequal treatment under the law, this university sued the Flemish authorities, and the Constitutional Court ruled that the Flemish government had to take action. Over the period 2005-2009, it was decided to increase the lump sum allocated to this university by 13 million euros.

In the period 2003-2006, additional funding of 15.5 million euros was also allocated to the academic bachelor’s and master’s programs organized by the university colleges to develop the integration of research and teaching in a stimulating partnership with the associated university.

Since 1991, universities have been strategic partners in the conception and implementation of the Flemish ambitious science and innovation policy agenda. They received a large share of the additional public R&D funding and legislation was implemented to stimulate the collaboration between universities and firms, the valorization of academic know-how and the creation of spin-offs.

Building on the above-mentioned initiative of the federal government at the end of the 1980s to provide additional research funding to universities, the Flemish authorities created a specific funding allocation for blue-sky research at the universities. The 1994 regulation stipulated that each university had to set up a dedicated internal funding mechanism for blue-sky research and complement the governmental allocation with a minimum percentage out of its own resources: the Special Research Fund or BOF under its Dutch acronym.

Annually, the Flemish parliament allocated a global sum to the BOF. Each university received a fraction of the global sum based on a formula taking into account the institution’s share in three input parameters: the size of the university’s lump sum, the number of master’s degrees it delivered, and the number of granted PhD degrees. The latter can also be considered as an intermediate parameter, as it is a measure of a university’s research capacity, but at the same time, it provides information on the training of the next generation of researchers. For each of these parameters a university’s fraction in the Flemish universities’ total value is calculated. In the partition formula to each parameter a weight factor was assigned, a reference period taken into account and averages made (Table 2).

The BOF became and still is the university’s instrument par excellence to develop its blue-sky research capacity based on internal selection procedures. The regulation stipulated that each university had to set up a research council, an internal advisory board to develop its research policy and to allocate the BOF-funding based on calls for proposals, and provide candidates with transparent evaluation procedures. The research council has a large autonomy, and over the years, it has gained within each university great prestige and legitimacy. The academic authorities had to approve the research council’s funding proposals, but their rubber-stamping became a mere formality.

Confronted with the need to set up their own research policy and lacking information on the strengths and weaknesses in different research disciplines, evaluating their research capacity became high on the Flemish universities’ agenda. In the 1990s, UGent was the first Flemish university that decided to carry out a systematic review of the research performance, followed by the KU Leuven and UAntwerp. These universities commissioned the Centre for Science and Innovation Studies (CWTS) at Leiden University to carry out this work. The methodology’s basic characteristic was the use of bibliometric indicators of research output and impact combined with a validation of the results by the researchers involved. The studies covered research groups in the natural and life sciences, medical and pharmaceutical sciences, mathematics, computer sciences, and engineering — disciplines where the publication culture is primarily focused on articles in international peer-reviewed journals (Korevaar and Moed, 1996; Moed et al., 1998; Luwel et al., 1999; Van den Berghe et al., 1998). In 1999, the Flemish Rector’s Conference commissioned a study on the development of indicators for research performance in the humanities and social sciences (HSS). This pilot study by CWTS was limited to law and linguistics (Nederhof et al., 2001).

The next anchor point in the development of performance-based funding metrics is the decision by the Flemish government to identify a small number of topics around which there was a need for policy-relevant research to proactively support policy development and develop the necessary academic research capacity. As science and innovation was one of the government’s priorities, in this call, a policy support center on quantitative science and technology studies located at the KU Leuven was selected. It became operational in 2003 and started collecting bibliometric and technometric data on the Flemish region.

In 2003, the Flemish government approved a major modification of the BOF-regulation: In the repartition of the global sum among the universities next to the above-mentioned cluster of three input parameters, a performance cluster composed of two bibliometric-based parameters was introduced (Debackere and Glänzel, 2004): publications processed for the Science Citation Index Expanded (SCIE) and assigned to the main publication types and citations to publications processed for the SCIE and assigned to these publication types. The former was considered a proxy of the university’s research capacity and the latter of its impact or visibility. Table 2 gives the weight assigned to each parameter.

The policy support center on quantitative science and technology studies was tasked with providing the data for the bibliometric-based indicators using a protocol that gave each university the possibility to verify the publication and citation data underlying the partition formula and, if necessary, suggest corrective actions.

The review of the regulation was planned, and a modified version had to go into effect in 2007, together with a revised funding formula for allocation of the lump sum.

In the explicative note annexed to the regulation, the government emphasized the experimental nature of the mechanism, its limitations, and the need to develop robust indicators for the HSS. Not taken into account, research in these disciplines turned out to be a sticky issue. Furthermore, the use of raw citation counts was considered unfair by many researchers and, given the differences in the citation culture between disciplines, in contradiction with the Flemish government’s objective to increase the international visibility of its blue-sky research.

But the first major revision of the BOF regulation in 2006 focused on the input parameters. Several issues related to the recruitment of tenured staff at the Flemish universities and the appeal to public authorities to intervene came to the center of the policy debate: the tendency to appoint in vacant tenured positions researchers who often spent their entire academic career at their alma mater and the underrepresentation of women in tenured positions. The government decided not to intervene directly in the autonomy of the universities but to use financial incentives.

In the 2006 revision of the BOF-regulation, minor technical changes to two of the three input parameters in the repartition formula were made, and a fourth related to mobility and diversity was added (Table 2).

In this regulation, the government also earmarked some of the additional BOF-funding to provide long-term support to world-leading researchers. The idea is similar to that of the Max Planck Institutions in Germany. Each university’s research council has to make the selection, and these researchers receive a substantial amount of funding until their retirement.

The collection and validation of the data used in the BOF partition formula remained of critical importance both for the acceptance by the universities and for the public authorities’ accountability. After an external review, the policy support center on quantitative science and technology studies was transformed into an inter-university center in 2007 and the development of a database on PhD degrees granted by the Flemish universities was added to its mission. The center also enlarged its bibliometric databases to all journals and proceedings editions of the Web of Science Core Collection.

As it was tasked with a long-term mission, in 2009, the Flemish parliament decided to grant it permanent status as an inter-university Centre for Research & Development Monitoring (ECOOM) with a twin mission: providing policy support and carrying out research on science and innovation.

The additional data from the Web of Science Core Collection were used in the 2008 revision of the BOF partition formula (Table 2). Starting in 2011, data from the Flemish database for the registration of publications in the HSS (VABB-SHW) were also included, and will be discussed in the next subsection.

In the input cluster, the major modification was the stepwise phasing out of the size of each university’s lump sum; starting in 2012, it was no longer taken into account.

The relative weight of the input cluster, on the one hand, and the performance cluster, on the other hand, slightly changed over the period 2007-2012, and from 2012 onward, the former weighed 64% and the latter 32% (Table 2).

During the years 2000-2012, the Flemish higher education system underwent a substantial reform. At the end of a long process in 2012, all academic degree programs except the artistic academic degree programs previously organized by the university colleges were integrated into the universities. The readable binary system with the universities offering academic-oriented degree programs and the university colleges’ professionally oriented bachelor’s programs was deemed necessary for the international attractiveness of the Flemish higher education system in the gradually developing European Higher Education Area (Degroof, 2018).

Even before this integration process, there were large discrepancies between the sizes of the Flemish universities. In the academic year 2012-2013, the last before the integration, KU Leuven and UGent together had 72% of the total university student population. Of this, 26%, roughly equally divided, enrolled in two median large universities—the UAntwerp and the VUB—and 1.7% in the UHasselt (Table 3). Due to this integration, the number of students enrolled at the Flemish universities increased by 34% between the academic years 2012-2013 and 2014-2015, the first year after the integration process was realized. In absolute numbers, the dichotomy between the two large universities and the other three became even more pronounced. However, the relative weight of the institutions remained fairly stable: KU Leuven and UGent together had 73%, with an increase of 3.3% of the former’s and a decrease by 2.6% of the latter’s share; the combined share of the UAntwerp and the VUB decreased by 2.2%, and the UHasselt was the big winner, nearly doubling its share.

As highlighted in the explicative note annexed to the 2012 revision of the BOF regulation, the supposed impact of this integration process on the three smaller universities’ research capacity was considered so important that radical measures were taken to protect their share in the BOF partition formula. For each of these institutions, a minimal fraction of the global sum allocated in the Flemish budget for the BOF was guaranteed.

In 2019, the Flemish government approved a new revision of the BOF regulation. The minimal fractions were confirmed, and until 2023, in the input cluster, each university’s fraction will no longer be calculated but fixed by the regulation. The performance parameters remained unchanged. A third cluster of six parameters was introduced in the formula. They are presented as proxies for governmental priorities (Table 2). Some parameters were in earlier versions included in the input cluster. Next to the parameter based on citations in the second cluster, in the third a new parameter using the citation distributions is introduced. The construction of this distribution is rather complex.

In third cluster a parameter for interdisciplinary research is announced. ECOOM is tasked with developing a methodology to measure interdisciplinary work at the Flemish universities and it will only be taken into account from 2024 onwards and at that moment the weights assigned to each parameter will be slightly modified.

As already mentioned, the use of Web of Science databases to construct performance indicators was severely criticized by a large part of the Flemish academic community. This approach was not only strongly biased toward the natural and life sciences and the basic disciplines in engineering, but it also gave at least the impression of neglecting the outcome and importance of scholarly work in the HSS. Its research culture is very different, and the Social Sciences Citation Index (SSCI) and the Arts & Humanities Citation Index (A&HCI) had low coverage of publications in these disciplines, especially those not written in English. In the 2008 revision of the BOF partition formula, the Flemish government took this criticism into account and tasked ECOOM to set up the VABB-SHW. The VABB-SHW includes the output of the Flemish universities in these disciplines that is not covered by the WoS. The ECOOM researchers at UAntwerp became responsible for managing this tool.

The 2008 revision of the BOF regulation specifies the publication types to be included in the VABB-SHW: articles in journals, monographs, edited books, articles or chapters in books and proceedings papers that are not part of special issues or edited books. A weight is assigned to each of the five publication types, with the weight of a monograph eight times higher than for a proceedings paper (Verleysen and Engels, 2018).

The Flemish government trusts a panel of influential scholars from the Flemish universities and university colleges working in these disciplines with the scientific management of the VABB-SHW. Based on the panel’s advice, the Flemish government laid down the procedure and detailed selection criteria. This selection is essentially done at the level of publishers and not of the individual publications.

The public authorities considered quality management as a critical success factor of this innovative policy tool. The regulation stipulates that every three years, an international panel should carry out a quality audit of VABB-SHW. The panel comprises at least five experts, of which one must be conducting research in science studies. The panel’s report with its conclusions and recommendations is sent to the Flemish government and parliament.

Flanders was not the first country or region to set up a bibliographic database to monitor the scientific publications of its universities. CRIStin, the Norwegian national research documentation system that was gradually developed by the universities from 1990 onward and became fully operational as a governmental management tool in 2010, is probably the best known and served as an inspiration for Flanders and two other Nordic countries, Denmark and Finland (Engels and Guns, 2018).

To develop a portfolio of strategic applied research, to stimulate collaboration with firms, and to promote the valorization of research results, in 2004, the Flemish government decided to grant each university earmarked funding for these activities. Strategic basic research is defined as applied research with a middle long-time horizon for economic or societal applications.

The regulation described in broad terms the objectives of this additional funding called the Industrial Research Fund or denoted with the Dutch acronym IOF. The only constraints were that the selection process had to be based on open calls and that at least 30% of the public funding had to be used for post-doc positions. It was stipulated that each institution could complement the public funding with proper resources such as revenues from its valorization activities. Each university had to develop its applied research and valorization strategy and report every five years on its objectives, outcomes, and adjustments. Although not mandatory, the regulation enabled the universities to set up the IOF funding jointly with the university colleges at the level of their association. To devise this strategy, to implement it, and to manage the dedicated funding, each university had to set up an IOF board. Representatives of the association’s university colleges and the business sector had to sit on this advisory body.

To determine each university’s fraction of the global sum annually allocated in the Flemish budget for the IOF, a partition formula was used. It was based on seven weighed parameters (Table 4). As with the BOF partition formula for each of these parameters, a university’s fraction in the Flemish universities’ total value is calculated, a reference period taken into account and averages made.

For publications and citations the data of the BOF partition formula were used; ECOOM provided the patent data, and for the other parameters, and the administration and the Rectors Conference made the data available. And as with the BOF partition formula for the data collected, a transparent procedure was used and the universities had access to all the data and could report on omissions or errors.

After a review, in 2009, the third parameter was replaced by each institution’s fraction in Flanders universities’ total revenues from research contracts with industrial partners, and from 2011 onwards, the parameter based on the number of tenured faculty and researchers was no longer taken into account. Some technical changes were made to the regulation, and the weight of the parameters was modified gradually over a three-year period (Table 4).

To incentivize an integrated strategy for strategic applied research and valorization in each association of university-university colleges, the parameters were no longer based on university but on association data. The 2009 revision of the regulation also made mandatory the organization of IOF funding at the level of the associations.

The regulation also stipulated that each association had to develop a five-year strategic plan for its strategic applied research and valorization activities and submit it to the Flemish government. As for all its funding instruments for science and innovation, the Flemish government commissions an external review of the IOF funding every five years.

In the 2014 revision of the IOF-regulation a major change was made in the repartition formula. As with the BOF regulation, from 2014 onward, a minimum fraction of the global IOF funding is guaranteed for the Association UAntwerp, the Association UHasselt, and the Association VUB. Together with slight changes in the weights, in the 2019 revision this modification was confirmed.

In 2008, the Flemish parliament approved a new funding formula for the lump sum of the universities and the university colleges. The model is rather complex. A detailed description and analysis are outside the scope of this contribution that focuses on its research components (See: De Cock and Vercruysse, 2009).

The main lump sum of each university has comprised four components:

To each of these four components, a global sum is allocated, and a partition formula is used. Each university receives base funding for education taking into account the number of awarded study-credits with some degree of degressivity. With the implementation of the Bologna Declaration study-credits were introduced as a standard for the degree programs (Huisman and Mampaey, 2017).

The funding formula for the partitioning among the universities of the global sum for the variable funding for education is, to a large extent, based on the awarded study-credits and the bachelor and master degrees delivered by each institution.

The distribution of the global sum for the base funding for research among the universities is based on the number of publications and granted PhD degrees. Both parameters are the same as those in the partition formula for the BOF.

To offset the effects on its research capacity of underfunding of UGent’s lump sum over the previous decades, it received a temporary increase of its base funding for research in the period 2008-2013.

The distribution of the global sum for the variable funding for research among the universities is also based on parameters of the 2008 version of the partition formula for the BOF:

In the period 2008-2013, a gradual transition was made from the existing to the new funding model. Only from 2014 onward and with a structural increase of the total amount allocated in the Flemish budget was the new funding formula used to calculate each institution’s lump sum.

The decree has a provision to adapt the variable funding for education to changes in the weighted number of study-credits. There is no equivalent provision for changes in the parameters used to determine the global sum allocated in the model to the variable funding of research.

First and foremost it should be emphasized that not only Flanders but also the other countries regularly update the funding formula. As illustrated below, these changes are often deemed necessary to remediate unintended consequences or short-circuit attempts to game the system. The underlying motives behind these modifications are an interesting research topic in itself.

The PRFSs are used to allocate research funding based on the universities’ relative performance in a zero-sum game. The frequency with which the PRFS is run differs among the countries. Models based on information collected at institutional level and peer assessment generally work with 3-5 year cycles and those relying on publicly available data have shorter cycles (Zacharewicz et al., 2019). Although it partially depends on information provided by the universities, the Flemish PRFS is implemented on a year-by-year basis. To avoid problems due to too strong fluctuations in allocated sums, as is the case in the Czech PRFS (Good et al., 2015), in the Flemish formulae long-term averages are applied.

Although publications play an important role in the ten PRFSs, the bibliometric indicators differ among the countries.4 They have all set up either a comprehensive quality assured current research information system (CRIS) or a quality controlled publication database to complement the WoS and/or SCOPUS. Some countries also include artistic creations (Poland (Kulczycki et al., 2017), Slovakia (Balasz, 2015)). Except Sweden5 that uses WoS data (Carlsson, 2009; Jacob et al., 2016), one of their functions is to provide the bibliometric data included in the PRFS.

In all the models differences in the publication culture among research fields are taken into account by applying elaborated field weighted normalizations of WoS or SCOPUS publication (and citation) data (e.g. the Swedish PRFS (Carlsson, 2009)) or by assigning weights or points to the scholarly work included in the CRIS. Researchers recognized as leaders in their discipline are tasked with the assessment of publishers and publications (e.g. the Norwegian PRFS (Sivertsen, 2018)). As the Flemish PRFS is based on the WoS and a complementary database for the HSS, both techniques are applied.

To complement the bibliometric criteria all 10 countries include in their PRFS a wide range of other indicators. Flanders and six other countries take into account the (weighted) number of granted PhD degrees.

All the countries use external funding as a criterion: competitive project funding from public national and/or international, mainly European Union’s research programs, and initiatives and/or revenues from research collaboration with the business sector. Countries use also criteria they consider important for their national policy agenda. As the strengthening of the national identity is one the Estonian government’s priorities, the participation in scholarly activities essential to sustain the Estonian language and culture is included in this indicator (Debackere et al., 2018). To develop the general public’s better understanding of the impact of science on everyday live, in the Croatian PRFS participation in activities related to the popularization of science is taken into account (Debackere et al., 2018).

Intellectual property rights and their valorization form the next cluster of indicators:

Flanders is one of the seven countries to include internationalization as a criterion in its PRFS. It is operationalized by the above mentioned participation in international research programs. Additionally Flanders (2019 revision of the BOF partitioning formula) and Norway (Debackere et al., 2018) give a higher weight to internationally co-authored publications; Finland takes into account both the PhD degrees granted to foreign researchers and the international teaching and research staff at its universities (de Boer et al., 2015); and Croatia uses information on incoming and outgoing mobility of the academic staff (Debackere et al., 2018).

Flanders is the only country to include since 2006 diversity as a criterion in the PRFS and it plans to introduce in the next couple of years a parameter quantifying interdisciplinary research activities.

Except for the latter two, the criteria used in the 10 countries’ PRFS are fairly similar aiming to strengthen the international competitiveness of university research and to enhance its contribution to the development of a competitive knowledge-based society.

While there is a fairly broad consensus on the criteria, among the countries there are large differences in the allocation of public research funding between:

For the Czech Republic, Denmark, Finland, Norway, Poland and Sweden, Reale (2017)6 gives an overview of the relative weight of these types of funding in the total public outlays for R&D. However no breakdown is given by institutional type, making it difficult to estimate these ratios for the university sector.

For the universities’ institutional research funding more information is available on the ratio between the performance-based and block funding. For the 10 countries this ratio differs between 2% (of the total institutional funding) for Norway (Aagaard, 2015) and 100% (of the institutional research funding) in the Czech Republic (Good et al., 2015). In Flanders the allocation of 50% of the BOF-funding and 75% of the IOF-funding is metrics-based and the allocation of the research component of the lump sum is fully metrics-based7 (De Cock and Vercruysse, 2009). Together they represent the institutional research funding of the Flemish universities. With 75% of the sum total allocated through PRFS, Flanders is among the countries with the highest ratio (EWI, 2020).

The different trade-offs made in the design of the PRFSs lead to the question of the realization of their objectives and their potentially unintended consequences on three levels: the national level as the mechanisms’ primary objective is the allocation of research funding among universities, the institutional level (university, faculty, department and research groups) and the individual researcher.

Over the last two decades empirical research has been done using two approaches: qualitative and quantitative. The former are based on document analysis, surveys and interviews with stakeholders. Although as shown in Table 5 in most PRFSs a variety of criteria are used, the latter are mostly limited to bibliometric studies of the publication output and impact.

In a pioneering study on performance-based funding in Australia, Butler (2003) showed that after the introduction of a PRFS based on publications covered by WoS aggregate national publication output increased but the country’s citation impact declined. This observation could have been one of the reasons the Australian government replaced it by a metrics-informed peer review assessment8 (Jonkers, 2016). However recently van den Besselaar et al. (2017) contested Butler’s observation leading to a discussion about assumptions and causality (Aagaard and Schneider, 2017; Butler, 2017; Gläser, 2017; Hicks, 2017). As already observed by Aagaard and Schneider (2016) in their analysis of the Danish system, this discussion highlights the difficulty to separate effects of the PRFS from other potential drivers in the science and innovation system and to identify causal relationships.

In a similar analysis of the long-term developments of the Norwegian publication activity, Schneider et al. (2016) observed an increase in publication output while the impact remained stable. The authors indicate that the introduction of the PRFS stimulated more research work and resulted in more publications, but no causal link could be made as during the same period R&D investments and the number of researchers increased also.

Ingwersen and Larsen (2014) analyzed the Danish publication output in the period 2000-2012, prior to and after the introduction of the PRFS in 2008. The number of journal articles covered by the CRIS grew during the whole period but progressed more steeply in the years 2008-2012. This growth rate outperformed the increase of academic staff. The number of articles processed for the WoS was also steadily increasing and the growth was even accelerating in 2010-2012. On the other hand, the number of the proceedings papers was nearly halved between 2007 and 2012. The average citation impact of all WoS items combined was linearly rising.

Korytkowski and Kulczycki (2019) analyzed the publication and patent output of the Polish universities and public research institutions during the period 2009-2016. The number of WoS publications increased without a decrease of the quality (in terms of JCR quartiles). It was found that this increase was not only due to a select group of researchers that became more productive, but that more researchers started publishing in journals indexed in the WoS. The authors link this result to the funding model and the stability over time in the assignment of weights to journal articles. Although during the same period the evaluation criteria of monographs were several times modified, their number remained stable.

Pisar et al. (2019) compared the number of Slovak publications in the WoS and SCOPUS between 2007 and 2016, four years before and five years after the introduction of the PRFS. During this period a significant increase in the publication output is observed. However, compared with Austria where after 2013 funding of universities was based on negotiations between the government and individual institutions, the publishing activity in both countries was fairly similar. Pisar et al. (2019) shows that the introduction of the PRFS increased the publication output in higher quality journals of universities that were initially less productive. But at the same time they provide evidence that less well performing universities also started gaming the rules to increase their output: launching new Central and Eastern European focused regional journals often published by the universities themselves which subsequently got indexed in WoS or SCOPUS and publishing in predatory journals that for a number of years managed to get indexed in these databases. To counter the tendency to increase output at all costs, the Slovak authorities started in 2017 to apply stricter rules and to take the journal impact factors into account.

The Czech PRFS is fairly similar to the Slovakian. Vanecek (2014) analyzed the publication output of the Czech universities and publicly funded research institutions in the period 2000-2011; the PRFS being introduced in 2009. The publication output (journal articles, books, book chapters and conference proceedings) covered by the Czech CRIS stagnated after 2007, due to a steep decrease of the number of contributions in conference proceedings. The subset of Czech publications indexed in the WoS grew rapidly between 2000 and 2010 followed by a small decrease in 2011. As for Norway, no substantial changes of the publishing journals’ impact factor were observed. The increase in the annual number of WoS-publications started in 2005 before the implementation of the PRFS. The growth in public R&D funding and/or the systematic evaluation of universities and research institutes in the period 2004-2008 could explain (at least partially) this trend. It should also be remarked that Pisar et al. (2019) formulated similar criticism on the Czech PRFS as on the Slovak claiming that a clear majority of the local and Eastern European journals indexed in the WoS or SCOPUS have only a ‘provincial status’ but they were taken into account in the allocation model. To avoid the overemphasis on local journals the Czech authorities are changing the PRFS combining a quantitative and qualitative approach.

For Flanders no similar studies are available. The biannual Flemish Indicator Book provides data on trends in the total Flemish publication output and impact. The number of WoS publications with a Flemish address per 10.000 inhabitants nearly doubled between 2006 and 2017 and the citation impact increased (Debackere and Veugelers, 2019). A large share of these publications is authored by researchers working at Flemish universities. However, no causal link can be made between the increase of the Flemish publication output and the use of the PRFS as the public research funding of universities and the number of researchers also considerably increased.

Compared to the WoS publication output, Flemish universities’ publications in the SSH have been comprehensively analyzed and benchmarked with other countries. For the period 2004-2015 Engels et al. (2018) studied trends in books and book chapters in the VABB-SHW and in the Finnish, Norwegian, Polish and Slovenian CRIS. For the humanities the share of monographs and book chapters in the peer reviewed publications indexed for VABB-SSH was fairly stable. Compared to Finland, Norway and Poland both shares were lower.

In Flanders’ peer-reviewed publication output in social sciences, the share of monographs was also fairly stable and increasing somewhat in 2014 and 2015 while share of book chapters was steadily increasing. The authors link the increase in book chapters to the introduction of the Guaranteed Peer Reviewed Content (GPRC) label (Verleysen and Engels, 2013). As for the humanities both shares were lower than for Finland, Norway and Poland.

For the HSS Ossenblok et al. (2012) compared trends in the coverage by WoS of publications included in the Flemish VABB-SHW and in the Norwegian CRISTin and the evolution of the share of articles published in English. Over the period 2005-2009 the fraction of peer reviewed journal publications of both countries written in English increased by 10%; in 2009 more than half were in English. Ossenblok et al. (2012) identified this trend as an indication of the growing internationalization.

Between 2005 and 2009 the fraction of publications in the VABB-HSW covered by the WoS increased steadily, while for Norwegian CRISTin it remained stable. For Flanders two factors could explain this trend. The tendency of Flemish scholars in SSH to publish more in WoS-covered journals and by the expansion of the WoS indexed more journals in these disciplines with local ties to Flanders and the Netherlands (Engels et al., 2012). The former could be related to the version of the BOF partition formula used up to 2008 that took only publications in journals indexed in the WoS into account. Contrary to the suggestion by de Rijcke et al. (2016), these results do not validate the claim that Flemish scholars’ quality considerations were displaced by incentives to increase their publication output. In fact, citations and impact were not analyzed by Ossenblok et al. (2012).

In a follow-up study Kulczycki et al. (2018) analyzed the publication types and languages used in the HSS in eight countries that have set up a CRIS. Although the coverage of these databases is not fully identical, broad patterns emerged. In 2014 79% of Flemish publications were written in English, which was the highest among the seven countries, followed by Finland (68%) and Denmark (63%). In the same year, 40% of the publications in these disciplines covered by the VABB-HSW were published in journals indexed in the WoS. Denmark (51%) and Norway (44%) have the highest share and Poland (15%) the lowest.

In quantitative studies of PRFSs little attention is paid to the other parameters used alongside bibliometric criteria. Patents and the valorization of IPRs are maybe the exception confirming the rule. Vanecek (2014) observed a strong increase since 2006 in patents and utility models granted by the national patent office to Czech universities and public research institutes. In the period 2002-2011 there was also an upward trend in the number of Czech patents granted by EPO and USTPO. Although no breakdown is given by sector, it seems reasonable to assume that the Czech universities have contributed to this increase. However, the introduction of the PRFS which was announced in 2008, could only partially explain this upward trend as the highest increase in the growth rate occurred already before. Moreover Good et al. (2015) makes some critical remarks about the role of national patents, utility models and prototypes granted by the Czech authorities. They should be easy to produce and no quality checks such as originality and functionality were required, yielding ‘cheaply earned’ points in the funding formula.

In the period 2009-2016 the number of patents submitted for the PRFS by the Polish universities and public research institutes increased until 2012 and then remained fairly stable (Korytkowski and Kulczycki, 2019).

The biannual Flemish Indicator Book (Debackere and Veugelers, 2019) provides information on Flemish patents and patent applications. Although the raw numbers are not published, the evolution of the fraction of the Flemish universities in the EPO patent applications with only Flemish applicants increased spectacularly: using two year averages from 3% in 1991-1992 to 12% in 1998-1999 and 18% in 2016-2017. Again this trend must be seen against the background not only of the introduction of the PRFS but also the strong growth in public research funding.

The ten countries use research funding from international, national and regional public authorities and/or from the private sector in their PRFSs. In the literature survey no study on project funding by public authorities and other R&D revenues for a country’s university sector or individual universities was found. Most probably these data are published in the local language in universities’ annual accounts, official documents, reports of commissioned studies and other grey literature.

Even for Flanders these data are not readily available. The only exceptions are the participation of Flanders at the different EU Framework Programs for Research and Innovation (FP) and the number of PhD-degrees granted by its universities.

Since 1991 for the successive FPs and by organization type the information on the number of projects and on the granted EU funding have been collected. Of the funding available for the 4^th FP (1994-1998) the Flemish universities received 1.01%; for the 5^th FP (2002-2006) 0.94%, for the 6^th FP (2002-2006) 0.86%, for the 7^th FP (2007-2013) 1.15% and for Horizon 2020, the current FP 1.12% (based on allocation of 58% of the available EU funding) (Debackere and Veugelers, 2019). As the overall success rate for obtaining FP funding is low, these data could be an indication of an increasing competitiveness of the Flemish universities. However, a more detailed benchmarking analysis is needed.

One of the tasks of ECOOM is the registration of information on doctoral researchers at the Flemish universities such as funding source, gender, success rate and time to degree (Groenvynck et al., 2013). Between the academic year 1991-1992 and 2017-2018 the number of awarded PhD-degrees increased more than fourfold to 2155. The share of women grew steadily to 45% (Debackere and Veugelers, 2019). Over the years the median time to degree is quite stable at 4.7 years but the doctoral completion rates have increased. Again these data have to be put in perspective as the public R&D funding grew also substantially during this period.

To complement these macro-level studies, for the ten countries there are only a few quantitative studies on the effects of PRFSs on meso- and micro-level. Bloch and Schneider (2016) analyzed the influence of the Norwegian PRFS at the individual level. In 2011-2012 compared to 2004-2005 a Norwegian researcher published more papers but at the same time the average number of coauthors per publication also increased. The lack of a clear definition of the publication activity of an individual researcher makes it challenging to link these results to the use of the PRFS.

Fiala (2013) analyzed the publication output of the Czech universities in the period 2008-2011. Although of each university both the total publication output in the CRIS and the fraction in journals indexed in the WoS increased, their relative weight in the Czech total (WoS) publication output changed to the detriment of the largest institutions.

Hammarfelt and de Rijcke (2015) studied research practices and publication patterns of scholars at the faculty of Arts at Uppsala University, Sweden using bibliometric data and a survey. Publication statistics were collected for the period 2006 to 2013 covering the introduction of both the Swedish PRFS in 2009 and the system for allocating resources based on performance measures at Uppsala University in 2011. Over the period the fraction of publications in English and of peer-reviewed publications increased. However the authors were not able to make a causal link with the implementation of the PRFS and/or the university level allocation model.

Sile and Vanderstraeten (2019) made a similar study on publication patterns within the field of educational research at the University of Gothenburg, Sweden during the period 2005-2014. This university introduced an institutional PRFS in the same year as the national one. The publication output rose, but the increase predated the introduction of the two PRFSs. Differences between publication types were observed: the number of peer-reviewed journal articles, especially those written in English grew rather steeply, but also the number of articles addressing a non-academic audience increased. The share of monographs remained stable.

For the Flemish model no similar studies have been published.

Although interesting, the above-mentioned publications focus on a single component of the PRFS. Comprehensive and international comparative quantitative analyses of the realization of all the objectives of the metrics-based PRFSs and their (un)desirable effects on the science and innovation system are necessary. Their result should be compared with those of other models for institutional research funding. They should go beyond the purely descriptive work commissioned by the EU ((Debackere et al., 2018), (Jonkers and Zacharewicz, 2016)). With the exception of Checchi et al. (2019), to the best of our knowledge no such studies have been done, let alone results published for scrutiny in the open literature. Using econometric techniques Checchi et al. (2019) examined to what extent the use of a PRFS for universities had an impact on their publication output and its quality. They found that the introduction of a PRFS increased a country’s publication output, however this result is only temporary. The effect of a peer review-based system is larger compared with a metric-based system. Using as an indicator for impact the number of publications in top journals the PRFS had no sizeable effect. One of the limitations of this study is its narrow focus on only one aspect of a university’s mission: publishing scientific results in the open literature.

Additional quantitative work could complement or serve as background material for qualitative research on the use of PRFSs and their implications on knowledge production, institutional management and (the careers of) individual researchers. Although based on the literature review of the much broader question of evaluation practices and effects of indicator use, de Rijcke et al. (2016) report that overall the knowledge of the implications of quantitative assessments and more broadly of how performance measures shape the production of knowledge is sketchy.

In the recent critical literature on the ‘state-of-the-art’ of institutional performance-based research evaluation arrangements, Thomas et al. (2020) identify limitations in our knowledge of their effects. Based on these observations and their belief that whatever the shortcomings performance-based funding arrangements are ‘here to stay’, the authors propose a novel research agenda to gain additional insights to support the development of more performant funding mechanisms.