The largest user study of large language models (LLMs) for assisting the general public in medical decisions has found that they present risks to people seeking medical advice due to their tendency to provide inaccurate and inconsistent information.
A new study from the Oxford Internet Institute and the Nuffield Department of Primary Care Health Sciences at the University of Oxford, carried out in partnership with MLCommons and other institutions, reveals a major gap between the promise of large language models (LLMs) and their usefulness for people seeking medical advice. While these models now excel at standardised tests of medical knowledge, they pose risks to real users seeking help with their own medical symptoms.
Key findings:
No better than traditional methods
Participants used LLMs to identify investigate health conditions and decide on an appropriate course of action, such as seeing a GP, or going to the hospital, based on information provided in a series of specific medical scenarios developed by doctors. Those using LLMs did not make better decisions than participants who relied on traditional methods like online searches or their own judgment.
Communication breakdown
The study revealed a two-way communication breakdown. Participants often didn’t know what information the LLMs needed to offer accurate advice, and the responses they received frequently combined good and poor recommendations, making it difficult to identify the best course of action.
Existing tests fall short
Current evaluation methods for LLMs do not reflect the complexity of interacting with human users. Like clinical trials for new medications, LLM systems should be tested in the real world before being deployed.
“These findings highlight the difficulty of building AI systems that can genuinely support people in sensitive, high-stakes areas like health,” said Dr Rebecca Payne, GP, lead medical practitioner on the study, Clarendon-Reuben Doctoral Scholar, Nuffield Department of Primary Care Health Sciences, and Clinical Senior Lecturer, Bangor University. “Despite all the hype, AI just isn’t ready to take on the role of the physician.
Patients need to be aware that asking a large language model about their symptoms can be dangerous, giving wrong diagnoses and failing to recognise when urgent help is needed.”
Real users, real challenges
In the study, researchers conducted a randomised trial involving nearly 1,300 online participants who were asked to identify potential health conditions and recommended course of action, based on personal medical scenarios. The detailed scenarios, developed by doctors, ranged from a young man developing a severe headache after a night out with friends for example, to a new mother feeling constantly out of breath and exhausted.
One group used an LLM to assist their decision-making, while a control group used other traditional sources of information. The researchers then evaluated how accurately participants identified the likely medical issues and the most appropriate next step, such as visiting a GP or going to A&E. They also compared these outcomes to the results of standard LLM testing strategies, which do not involve real human users. The contrast was striking; models that performed well on benchmark tests faltered when interacting with people.
They found evidence of three types of challenge:
“Designing robust testing for large language models is key to understanding how we can make use of this new technology,” said lead author Andrew Bean, a doctoral researcher at the Oxford Internet Institute. “In this study, we show that interacting with humans poses a challenge even for top LLMs. We hope this work will contribute to the development of safer and more useful AI systems.”
“The disconnect between benchmark scores and real-world performance should be a wake-up call for AI developers and regulators”, said senior author Dr Adam Mahdi, Associate Professor, Reasoning with Machines Lab (OxRML), at the Oxford Internet Institute. “Our recent work on construct validity in benchmarks shows that many evaluations fail to measure what they claim to measure, and this study demonstrates exactly why that matters. We cannot rely on standardised tests alone to determine if these systems are safe for public use. Just as we require clinical trials for new medications, AI systems need rigorous testing with diverse, real users to understand their true capabilities in high-stakes settings like healthcare.”
Download a copy of the study, ‘Reliability of LLMs as medical assistants for the general public: a randomized preregistered study‘, published by Nature Medicine.
Notes for Editors
Contact
For more information or to arrange interviews with the lead authors, please contact: Sara Spinks / Veena McCoole, Media and Communications Manager.
M: +44 (0)7551 345493
Media spokespeople:
About the research
This study focused on the ability of large language models like ChatGPT to provide medical advice to people at home. The study used a controlled trial to test the usefulness of large language models as sources of medical advice. Research included 1,298 online participants, selected to represent the demographics of the UK adult population.
The authors of the study are:
Funding information
This research was supported by funding from Prolific and the Oxford Internet Institute’s Research Programme on AI, Government, and Policy, funded by the Dieter Schwarz Stiftung GmbH. The Data-centric Machine Learning Working Group at MLCommons provided support in using the Dynabench platform. Additional support was provided by the Royal Society Research Grant RG\R2\232035 and the UKRI Future Leaders Fellowship grant MR/Y015711/1, and the National Institute for Health Research (NIHR) Oxford Biomedical Research Centre (BRC). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.
About the Oxford Internet Institute (OII)
The Oxford Internet Institute (OII) has been at the forefront of exploring the human impact of emerging technologies for 25 years. As a multidisciplinary research and teaching department, we bring together scholars and students from diverse fields to examine the opportunities and challenges posed by transformative innovations such as artificial intelligence, large language models, machine learning, digital platforms, and autonomous agents.
About the University of Oxford
Oxford University has been placed number one in the Times Higher Education World University Rankings for the tenth-year running, and number two in the QS World Rankings 2022. At the heart of this success are the twin-pillars of our ground-breaking research and innovation and our distinctive educational offer. Oxford is world-famous for research and teaching excellence and home to some of the most talented people from across the globe.
DPhil Student
Andrew holds a B.S. in Applied Mathematics from Yale University and an MSc in Social Data Science from the OII. He is a Clarendon Scholar and was previously a Thouron Prize winner at the University of Cambridge (Pembroke College).
Associate Professor
Luc conducts human-centred computing research to understand how data and algorithms impact society. They work to make digital power visible to the public and guide the development of accountable, sustainable, and safe algorithms for all.
Associate Professor
Adam Mahdi’s research focuses on digital health and application of machine learning in social sciences. He is the director of the UKRI-funded OxCOVID19 Project and a fellow at Wolfson College, University of Oxford.
Clarendon-Reuben Doctoral Scholar and General Practitioner, Nuffield Department of Primary Care Health Sciences and Bangor University
Rebecca is a General Practitioner, Chair, NICE Quality Standards Committee, Honorary Clinical Senior Lecturer, Bangor University and Churchill Fellow.
DPhil Student
Guy is a Shirley Scholar at the OII, an Intensive Care doctor, and the Clinical Lead for a Health AI business at Deloitte. He researches AI-enabled healthcare systems.
DPhil Student
Hannah is a 4th year DPhil student in Social Data Science at the OII and a Research Scientist at the UK AI Security Institute. Her research explores how to align AI systems with the values and preferences of diverse human populations.
Our work investigates applications of large language models (LLMs) in healthcare settings, with a particular focus on interactions between LLMs and human users. The project focuses on LLMs for medical self-diagnosis.
