OpenAI’s ChatGPT passes the United States Medical Licensing Exam

OpenAI's ChatGPT has “performed at or near the passing threshold” on the United States Medical Licensing Exam (USMLE), researchers have written, adding that it demonstrated a high level of concordance and insight in its explanations to the answers.

Researchers tried ChatGPT’s performance on sitting the United States Medical Licensing Exam. The three-part exam is usually taken by aspiring doctors between medical school and residency.

Reporting their findings in a paper published in December 2022, the researchers noted that ChatGPT “performed at or near the passing threshold for all three exams without any specialised training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations.”

ChatGPT has been trained on large language models and the feedback shows they may have “the potential” to be of help in medical education and even clinical decision-making, the abstract noted.

The Insider, which reported on the results, wrote the research was still under peer review.

The chatbot is also said to have demonstrated “high internal concordance,” with its answers and explanations scoring 94.6 percent concordance across all questions as reviewed by two physician adjudicators.

“Our findings can be organised into two major themes: (1) the rising accuracy of ChatGPT, which approaches or exceeds the passing threshold for USMLE; and (2) the potential for this AI to generate novel insights that can assist human learners in a medical education setting,” the researchers wrote.

ChatGPT has already been used by students to write essays, which prompted Bloomberg podcaster Matthew S. Schwartz to proclaim in December the death of the "take home essay”.

On another occasion, a student was caught by a philosophy professor at Furman University caught turning in an AI-generated essay upon noticing it had "well-written misinformation," Insider reported.

“Word by word it was a well-written essay,” the professor told Insider. However, upon closer scrutiny, he noticed that the student made a claim about the philosopher David Hume that “made no sense” and was “just flatly wrong”, Insider reported.

Among other feats that ChatGPT has recently pulled off are passing exams in four law school courses at the University of Minnesota, busting Alex Berezow’s Big Think 10-question microbiology quiz and passing Wharton professor Christian Terwiesch’s final exam in operations management.