Using Large Language Models for Research Evaluation in the REF

Evaluating the quality of research outputs is time consuming and difficult, not just for the 1000+ REF evaluators but also for internal departmental REF selection procedures. In REF2021, 11 out of 34 Units of Assessment consulted citation-based indicators to support their decisions and they may also have been used by departments selecting their best outputs. Nevertheless, it has recently emerged that ChatGPT can give stronger evidence than citation-based indicators in most fields. This has created the possibility to extend the use of indicators to support research quality judgements to more fields and perhaps also in a slightly stronger role. This talk will describe the results of experiments that have shown this capability. Frustratingly from a qualitative perspective, despite statistically significant score predictions by ChatGPT and plausible reports about rigour, originality and significance, its evaluations seem to be too imprecise to provide strong evidence of its predicted scores. It is not clear whether this is a large language model limitation or a more fundamental property of academic expert review. ChatGPT’s other quirks include a liking for post-humanism, being unimpressed with articles in The Lancet, and being willing to hallucinate detailed evaluations of non-existent papers.

Mike Thelwall is a social scientist and professor of data science at the University of Sheffield, specialising in research evaluation. He has advised the REF on the selection and use of citation-based indicators and is consulting about a potential transition to large language models. He is a senior associate editor of the Journal of the Association for Information Science and Technology, and his current book, Quantitative Methods in Research Evaluation Citation Indicators, Altmetrics, and Artificial Intelligence, is free at https://arxiv.org/abs/2407.00135.