Some AI models designed to predict stroke and diabetes risk may be based on datasets whose origins cannot be verified, according to new research.
The study, published in BMC Medicine and led by researchers at QUT and the Australian Centre for Health Services Innovation (AusHSI), examined two widely downloaded health datasets hosted on Kaggle, an online platform for sharing datasets and machine-learning resources, marketed as “the world’s AI proving ground”.
The datasets were found to have been used in 125 peerreviewed studies, despite providing almost no information about where the data came from, how it was collected or whether it represented real patients.
Lead author Alexander Gibson, from the QUT School of Public Health and Social Work and AusHSI, said the team was shocked by what they found.
“It was an enormous surprise to come across something like this,” Mr Gibson said.
“These datasets exhibit unusual patterns that raise serious questions about their authenticity and suitability for clinical research.”
Three prediction models based on the data had evidence of use in clinical practice, one model was cited in a medical device patent, and the models were cited in 86 review articles.
The study assessed the datasets using the internationally recognised TRIPOD+AI reporting framework and found that they scored 0 out of 9 on essential dataprovenance criteria.
Mr Gibson said this should be a red flag for journals, developers and clinicians.
“Prediction models built on data of unknown provenance have no place in clinical decision-making. Without trustworthy data, the outputs are unreliable and risk misleading clinicians and harming patients,” he said.
The authors said that journals, funders and data repositories must strengthen requirements for datasource disclosure.
They also recommended that the two Kaggle datasets be removed to prevent further misuse.
Seven articles using these datasets have been retracted from journals for being unreliable. The results of study have also updated the Collection of Open Science Integrity Guides.
Mr Gibson said the issue reflects a broader challenge as AI tools proliferate in health care.
“We’re seeing fastchurn research built on datasets that look scientific but lack the most basic transparency,” he said.
“Without stronger safeguards, unreliable models will continue to make their way into the literature, and potentially into practice.”
The study also included QUT researchers Professor Adrian Barnett and Associate Professor Nicole White.
Read the full study, Evidence of Unreliable Data and Poor Data Provenance in Clinical Prediction Model Research and Clinical Practice, published in BMC Medicine, online.
Main photo (left to right): Professor Adrian Barnett, Alexander Gibson, Associate Professor Nicole White.
