Process for reforming research evaluation

First European Agreement "Broadening the spectrum of what is valued in research, in order to maximise the quality and impact of research".

Given the evolution of the international movement promoting the need to reform research evaluation practices, the European Commission, between March and November 2021, promoted a highly participative process of consultation with all those actors with competences in research evaluation, i.e. both evaluating bodies and research organisations, as well as those funding or executing research. All this has crystallised in an Agreement on Research Assessment Reform (ARRA, 2022), published on 20 July 2022 and officially opened for signature on 28 September of the same year, which is the result of a co-creation process involving more than 350 organisations from 40 countries. To carry out this reform, a worldwide Coalition of Organisations for the Advancement of Research Assessment (CoARA) has been formed, which ANECA joined in April 2023.

It clearly states the intention to design a common direction, always respecting the autonomy of the organisations, as well as their specific frameworks and contexts, but which, in any case, aims to make a notable effort to recognise the diversity of missions or objectives, careers, profiles, results, processes, practices and activities that are included in research activity today, with the ultimate aim of maximising the quality and impact of this research.

The agreement is structured around a set of principles defining general conditions and those to be taken into account for evaluation criteria and processes, as well as a set of ten commitments to be fulfilled by adhering organisations. According to the proposed timetable for reform, any committing organisation will have to set out its action plan with defined milestones up to 1 year after signing the agreement and, consequently, will have to undergo a first review to show progress after 5 years.

Principles for general conditions
Comply with standards and practices of ethics and integrity. Safeguard scientific freedom. Respect the autonomy of research organisations. Ensure the independence and transparency of data, infrastructures and criteria used to assess research and determine its impact.

Principles for research criteria and processes
Quality and Impact	Diversity, Inclusion and Collaboration
Focusing research criteria on quality, which means rewarding originality of ideas, as well as professional conduct and results that go beyond the state of the art. It also means recognising a variety of research missions, methodologies, verifiable and reproducible results, data sharing and open collaboration. It is also noted that evaluation should be based on qualitative judgements for which peer review is essential, supported by quantitative indicators, used responsibly and where appropriate. Recognise contributions that advance knowledge, as well as the (potential) impact of their results, which may be of a very different nature (scientific, technological, economic and/or social).	Recognise the diversity of research activities and practices, of results, encouraging the early exchange of knowledge and data and collaboration open to very different actors. Use assessment criteria and processes adapted to the different profiles to which they apply: according to the characteristics of each discipline, according to the different stages in the research career, according to multi-, inter- and trans-disciplinary approaches, etc. Valuing diversity of roles and careers in research; as well as team science and collaboration. Ensure gender equality, equal opportunities and inclusion.

Principles for research criteria and processes

Quality and Impact

Diversity, Inclusion and Collaboration

Focusing research criteria on quality, which means rewarding originality of ideas, as well as professional conduct and results that go beyond the state of the art. It also means recognising a variety of research missions, methodologies, verifiable and reproducible results, data sharing and open collaboration. It is also noted that evaluation should be based on qualitative judgements for which peer review is essential, supported by quantitative indicators, used responsibly and where appropriate.
Recognise contributions that advance knowledge, as well as the (potential) impact of their results, which may be of a very different nature (scientific, technological, economic and/or social).

Recognise the diversity of research activities and practices, of results, encouraging the early exchange of knowledge and data and collaboration open to very different actors.
Use assessment criteria and processes adapted to the different profiles to which they apply: according to the characteristics of each discipline, according to the different stages in the research career, according to multi-, inter- and trans-disciplinary approaches, etc.
Valuing diversity of roles and careers in research; as well as team science and collaboration.
Ensure gender equality, equal opportunities and inclusion.

Core Commitments

Support Commitments

Recognise the diversity of contributions as well as roles and careers in research, depending on the objectives and nature of the research.
Base the evaluation of research to a greater extent on qualitative methods, recovering the fundamental weight of peer review and basing these judgements on quantitative methods, making responsible use of these indicators where appropriate. The efforts that this approach requires should be duly recognised in the careers of the professionals involved.
Abandon the inappropriate use of journal and publication-based metrics; in particular, the inappropriate use of the Journal Impact Factor (JIF) and the H-index.
Avoid the use of international rankings of research organisations, as their metrics are inadequate to evaluate researchers.

Allocate the necessary resources to carry out this reform of research evaluation, thus enabling the organisational changes that have been committed to be achieved.
Review the current evaluation criteria, tools and processes and develop new ones that are relevant, i.e. adapted to the different contexts in which they will be applied. These are the units and institutions, as well as the projects and researchers, and always in accordance with the established principles. To this end, all actors involved should work closely together.
Raise awareness of the reform through communication, guidance and training on the criteria and processes designed, as well as on their use, to all the agents involved.
Encourage the exchange of practices and experiences to enable mutual learning, both within and outside the Coalition. Avoid fragmentation, promote coherence and thus the mobility of researchers.
Communicate progress made in adhering to the Principles and implementing the Commitments. Self-reflection, monitoring, sharing successes and challenges; thus facilitating collective progress.
Evaluate the practices, criteria and tools developed, based on sound evidence and making publicly available the data used to do so.

National Chapter

In line with the Agreement and following its accession to the Coalition, Spain presented its proposal for a National Chapter, which is jointly coordinated by the Spanish National Agency for Quality Assessment and Accreditation (ANECA), the Conference of Rectors of Spanish Universities (Crue) and the Spanish National Research Council (CSIC) and brings together more than seventy universities, research centres, national quality assurance agencies and scientific societies.

Its objectives are closely related to the commitments of the Agreement, with special attention to commitments 6, 7 and 8, relating to the review and design of new criteria, tools and processes for research evaluation, to the commitment with entities linked to this area, whether they have signed the agreement or not, and to mutual learning, respectively.

As a concrete work plan, derived from the initiative that took place at the beginning of 2023, the Forum for the Reform of Research Assessment in Spain, promoted by the Ministry of Universities, and the main concerns that emerged from it, 4 specific developments are proposed, which will consist of the following:

Firstly, to identify best practices at international, European and national level, which can serve as inspiration for the development of new practices.
Secondly, to analyse these practices in relation to the national context, in order to propose new evaluation models, which will be tested in the format of pilot projects.
Thirdly, reports will be produced on the results of these pilots.
Fourthly, a final evaluation of the first two years of implementation will be carried out, with the aim of delivering a report containing the main progress achieved and also identifying the next steps (challenges).

For its part, ANECA already integrates, in a flexible manner, the orientations of this international movement of research evaluation reform in two of its evaluation programmes, both in the evaluation of research periods (Sexenios - Call 2023; especially in the Resolution of evaluation criteria and in the Appendix of new dimensions, sources and metrics, as well as in the proposed general scale, where the evaluation criteria are specified; and in the State Accreditation for university teaching bodies. For the first time, a process of information and consultation of the research evaluation criteria has been opened and the coherence of the merits and criteria used in the Agency's various evaluation procedures will be guaranteed.

Highlights

Among the most important highlights of this long journey, the following references can be highlighted in the international context:

2012: Dora(San Francisco Declaration on Research Assessment)-ANECA signed up in April 2023: There is a pressing need to improve the way in which funding agencies, academic institutions and other groups assess scientific research.
2014: Leiden (Manifesto on research indicators) - ANECA signed up in October 2021: There is no one evaluation model that can be applied to all contexts. The objectives of a research programme have to be specified at the outset and the indicators used to measure performance have to be clearly related to these objectives. Research functions and evaluation objectives change or shift and the research system co-evolves with them. Indicators have proliferated: usually well-intentioned, not always well-informed and often poorly applied.
2020: Los Principios de Hong Kong for the evaluation of researchers: Promoting research integrity: Five principles are presented, focusing on the need to drive research improvement by ensuring that researchers are explicitly recognised and rewarded for behaviours that strengthen research integrity.
2021: UNESCO (UNESCO Preliminary Draft Recommendation on Open Science): Making scientific knowledge, methods, data and evidence freely available and accessible to all increases scientific collaborations and information exchange for the benefit of science and society, and opens up the process of scientific knowledge creation and dissemination to societal actors outside the institutionalised scientific community. All this clearly contributes to the efficiency and performance of the system, increasing the likelihood of responding effectively to today's challenges.
2022: CoARA (Coalition of Organisations for the Advancement of Research Evaluation) - ANECA joined in April 2023: We will work together to enable systemic reform based on common principles within an agreed timeframe, and to facilitate information exchange and mutual learning among all those who wish to improve research evaluation practices.

Already within the national context, there is an important framework that reflects the spirit of the agreement, as well as important developments already in this context, such as:

2022: LCTI (Law 17/2022, of 5 September, amending Law 14/2011, of 1 June, on Science, Technology and Innovation): Research results will be available in open, free and open access, for which institutional or thematic repositories will be promoted.
2023: LOSU (Ley Orgánica 2/2023, de 22 de marzo, del Sistema Universitario Ley 2/2023): Scientific knowledge shall be considered a common good, accessible and non-commercialised.
2023-2027: ENCA (National Open Science Strategy, May): Axes: A. Digital infrastructures; B. Data management; C. Open access to publications; and D. Incentives and training. Data management; C. Open access to publications; and D. Incentives and training.
2023: RD 678/2023 (Royal Decree on State Accreditation, 18 July): Aneca must guarantee the coherence of the merits and criteria used in its various assessment procedures. It aims to improve the quality of assessment by including a greater plurality of methods, criteria and assessed contributions; as well as a greater contextualisation of the assessment.
2023: CNEAI (Resolution of 5 December, of the National Commission for the Evaluation of Research Activity, which publishes the criteria for the evaluation of research activity): And this takes the form, in accordance with the rules applicable to each scientific discipline, of a request for a narrative justifying the evidence of the relevance and impact of each contribution, supported by a responsible use of quantitative indicators.
2023: Capítulo nacional: Spain (ANECA-CRUE-CSIC, 2023): The European reform initiative opens an opportunity to reflect on the research evaluation system in Spain and will allow proposals for improvement to be put forward in line with the rest of the international research community, so that research and innovation in Spain continue to grow in excellence and academic and social impact.