Initial findings from the PEI3AC project on new academic assessment methodologies

• The Agency brings together the assessors involved in a series of experiments to jointly analyse the evidence gathered and discuss future improvements to the assessment processes

On 14 July, ANECA held a workshop dedicated to the presentation and discussion of the preliminary results of the PEI3AC project (Pilot Experiments for Innovative Academic Assessment under ANECA’s CoARA Action Plan), an initiative funded by the European programme CoARA Boost – Cascade Funding to explore, evaluate and validate innovative academic assessment methodologies. The meeting brought together, at the Agency’s headquarters, the assessors who have taken part in the experiments carried out in March, with the aim of sharing lessons learnt, comparing results and reflecting on possible improvements to evaluation processes.

PEI3AC was one of the projects selected in the second call for funding launched by CoARA, as part of the Horizon Europe programme, aimed at promoting institutional changes and new research evaluation practices. In the case of ANECA, the project has been carried out in the context of the implementation of the new state accreditation procedure regulated by Royal Decree 678/2023 and the commitments undertaken by the Agency regarding the reform of research evaluation.

The initiative involved the voluntary participation of 47 evaluators from the ACADEMIA programme, drawn from different fields of knowledge, universities and regions, representing 42 disciplines, 22 accreditation committees and 15 autonomous communities, with a composition that was equal and balanced in terms of gender and evaluation experience.

The project was structured around three days of experiments designed to address some of the main challenges of academic evaluation. The first focused on the possible effect of different ways of presenting curriculum information, comparing disaggregated narrative formats with more aggregated or synthetic formats. The second analysed the role of quantitative indicators versus qualitative descriptions of scientific contributions. The third day explored the possible existence of unconscious biases associated with non-traditional academic profiles by evaluating career paths with different personal and professional characteristics.

In addition to quantitative and qualitative findings, the project has provided valuable information on assessment practices and deliberative processes. Key preliminary findings include the widespread acceptance of the narrative component as an assessment tool, the importance of continuing to strengthen the training and guidance of assessors, and the role played by forums for collective discussion in promoting consistency in decision-making and shared reflection on assessment criteria.

Today’s session marks a new phase of the project, focusing on the dissemination and discussion of the results with those who have participated directly in the experiments: the dynamics observed in the individual and collective assessment processes were analysed, and working groups were set up to identify proposals for improvement relating to curriculum formats, assessment support tools, guidance for diverse profiles, and the functioning of assessment committees.

Through this initiative, ANECA continues to advance its strategy for reforming academic evaluation, based on empirical evidence and the active participation of the evaluation community. The results of PEI3AC will help to strengthen existing knowledge about how evaluation processes actually work and will support future methodological improvements, both within the Agency and within the European context of research evaluation reform led by CoARA.