Researchers from the Collegium Medicum of the Nicolaus Copernicus University put ChatGPT through its paces by testing its knowledge of disease treatment and diagnosis.
The artificial intelligence chatbot, however, did not make the grade, failing an internal medicine exam designed by the scientists.
While AI has made significant strides, including in the medical field, where it aids in tasks such as designing drugs, assisting in diagnoses, predicting pandemics, and even substituting for surgeons, the recent study by experts from the Ludwik Rydygier Collegium Medicum in Bydgoszcz advocates caution.
Marcin Behrendt reported via the Nicolaus Copernicus University website that the study, published in the Polish Archives of Internal Medicine, underscores the continued necessity of human expertise, particularly in internal medicine.
While AI has made significant strides, including in the medical field, where it aids in tasks such as designing drugs, assisting in diagnoses, predicting pandemics, and even substituting for surgeons, the recent study by experts from the Ludwik Rydygier Collegium Medicum in Bydgoszcz advocates caution.
Marcin Behrendt reported via the Nicolaus Copernicus University website that the study, published in the Polish Archives of Internal Medicine, underscores the continued necessity of human expertise, particularly in internal medicine.
The study indicates that internal medicine demands extensive knowledge, focus, and discipline. The assessment process, which includes a written test with multiple-choice questions and an oral examination, underscores the rigorous standards set for internal medicine specialists.
Despite ChatGPT’s success in other medical exams globally, including the United States Medical Licensing Examination (USMLE) and the European Exam in Core Cardiology, the Polish scientists conducted the first-ever evaluation of AI in the field of internal medicine.
ChatGPT was presented with over a thousand questions from Polish board certification exams spanning several years. However, its performance fell short, with correct answer rates ranging from 47.5% to 53.33%, well below the 60% required to pass.
Notably, ChatGPT struggled more with longer questions and those of higher difficulty, mirroring human behavior.
The study also assessed ChatGPT’s proficiency across various medical specialties, revealing varying levels of accuracy, with the model performing best in allergology (the study of the treatment of allergies) and least effectively in cardiology.
“Internal medicine, as a field, is often referred to as the queen of medical science. Physicians specializing in internal medicine are required to possess extensive knowledge as well as a high degree of focus and self-discipline,” the authors of the study published in the Polish Archives of Internal Medicine stated.
“Undoubtedly, it is worthwhile following the development of AI, especially ChatGPT, to be able to take advantage of its rapid progress,” the researchers highlighted.
“It is unlikely that AI will be able to replace health care professionals in the near future, particularly in the field of internal medicine; even the most sophisticated algorithms and technologies facilitated by AI are incapable of diagnosing and treating diseases without human input,” the study’s authors concluded.
Despite ChatGPT’s success in other medical exams globally, including the United States Medical Licensing Examination (USMLE) and the European Exam in Core Cardiology, the Polish scientists conducted the first-ever evaluation of AI in the field of internal medicine.
ChatGPT was presented with over a thousand questions from Polish board certification exams spanning several years. However, its performance fell short, with correct answer rates ranging from 47.5% to 53.33%, well below the 60% required to pass.
Notably, ChatGPT struggled more with longer questions and those of higher difficulty, mirroring human behavior.
The study also assessed ChatGPT’s proficiency across various medical specialties, revealing varying levels of accuracy, with the model performing best in allergology (the study of the treatment of allergies) and least effectively in cardiology.
“Internal medicine, as a field, is often referred to as the queen of medical science. Physicians specializing in internal medicine are required to possess extensive knowledge as well as a high degree of focus and self-discipline,” the authors of the study published in the Polish Archives of Internal Medicine stated.
“Undoubtedly, it is worthwhile following the development of AI, especially ChatGPT, to be able to take advantage of its rapid progress,” the researchers highlighted.
“It is unlikely that AI will be able to replace health care professionals in the near future, particularly in the field of internal medicine; even the most sophisticated algorithms and technologies facilitated by AI are incapable of diagnosing and treating diseases without human input,” the study’s authors concluded.
Source: Science in Poland
More In Society MORE...
