Comparative Study: ChatGPT vs. Human Evaluators Assessing Medical Literature

UK: In the face of clinicians’ ongoing challenges in keeping up with the ever-expanding body of medical research, artificial intelligence (AI) tools have emerged as potential solutions. One such tool, ChatGPT3, a large language model, is being assessed for its capability to automate the appraisal of research quality, with the aim of saving time and reducing bias in the process. 

Researchers from Swansea University, UK, have conducted a comparative study to determine ChatGPT’s proficiency compared to human evaluation when it comes to scoring research abstracts according to recognised reporting standards.

Methodology: Comparing ChatGPT and Human Evaluators

To assess the effectiveness of ChatGPT in this context, the researchers compared ChatGPT’s scoring of implant dentistry abstracts against human evaluators. They employed the Consolidated Standards of Reporting Trials for Abstracts (CONSORT-A) reporting standards checklist to establish an overall compliance score (OCS). 

ChatGPT’s Accuracy in Citing Journal Articles Under Scrutiny

The study also delved into a Bland-Altman analysis to evaluate the agreement between human evaluators and the AI-generated OCS percentages. Additional error analysis considered the mean difference of OCS subscores, Welch’s t-test, and Pearson’s correlation coefficient.

Findings: ChatGPT’s Performance and Correlations

The results of the study are revealing. Bland-Altman analysis revealed a mean difference of 4.92% (95% CI 0.62%, 0.37%) in OCS between human evaluation and ChatGPT. 

In the error analysis, there were only slight mean differences across most domains, with the highest observed in the ‘conclusion’ domain (0.764 (95% CI 0.186, 0.280)) and the lowest in ‘blinding’ (0.034 (95% CI 0.818, 0.895)). Notably, strong correlations were observed in the ‘harms’ (r=0.32, p<0.001) and ‘trial registration’ (r=0.34, p=0.002) domains, while weaker correlations were found in the ‘intervention’ (r=0.02, p<0.001) and ‘objective’ (r=0.06, p<0.001) domains.

ChatGPT and the Future of Dentistry

Implications and Future Prospects

The study’s conclusion highlights the potential of large language models like ChatGPT in automating the appraisal of medical literature, thereby contributing to the identification of accurately reported research. 

Potential applications of ChatGPT include its integration within medical databases for abstract evaluation. However, it is crucial to acknowledge the current limitations, particularly the token limit, which confines its use to abstracts. As AI technology continues to advance, future iterations like GPT4 could offer more reliable and comprehensive evaluations. 

This, in turn, could significantly enhance the identification of high-quality research, potentially leading to improved patient outcomes.