The text robot ChatGPT would do remarkably well in an exam for prospective medical students in the USA. Under certain conditions, the AI software was able to achieve the required minimum number of points in the three theoretical parts of the United States Medical Licensing Exam (USMLE). This is reported by US experts in the journal “PLOS Digital Health”. However, the team led by Victor Tseng from the Californian startup AnsibleHealth left out a few questions.
The USMLE is a standardized three-part exam that medical students must pass in order to practice medicine in the United States. Knowledge from most medical disciplines – from biochemistry to diagnostic thinking to bioethics – is evaluated.
However, when testing with ChatGPT, the authors of the study had to take into account that the OpenAI AI software can only accept text input. After removing image-based questions, the authors tested the software on 350 public-facing questions that were part of last summer’s USMLE exam.
After ambiguous answers were also removed, ChatGPT achieved 52.4 to 75 percent of the points that could still be achieved in the three USMLE parts of the exam. The results varied depending on the part of the test and the type of task – such as tick tests or open questions. The threshold for passing is around 60 percent and changes slightly depending on the year. If the ambiguous answers were included in the result, ChatGPT achieved 36.1 to 61.5 percent of the possible points.
According to the authors, ChatGPT outperformed PubMedGPT, a counter-model trained exclusively on biomedical literature.
The team concluded that ChatGPT had the potential to improve medical education and, in turn, clinical practice. “Achieving the pass mark on this notoriously difficult expert exam, and doing so without any human assistance, is a remarkable milestone in the development of clinical AI,” the authors wrote.
The text robot ChatGPT had previously asserted itself in other university subjects, even if it did not achieve top marks in the exams. According to professors at the University of Minnesota, the AI chatbot tool passed the law exams in four courses and another course at the Wharton School of Business at the University of Pennsylvania.
After the public launch of ChatGPT by the Californian start-up company OpenAI in November 2022, a number of players in the tech industry were in a real race for the favor of users. OpenAI is supported by Microsoft with billions of dollars.
In return, Microsoft may integrate the technology into its Bing search engine and other products. The big opponent of OpenAI and Microsoft is Google. The search engine giant launched its own chat bot Bard on Tuesday. Smaller players such as You.com by German AI researcher Richard Socher are also taking part in the race. Socher comes from Dresden, but has been living and working in the USA for years.
“Aha! Ten minutes of everyday knowledge” is WELT’s knowledge podcast. Every Tuesday and Thursday we answer everyday questions from the field of science. Subscribe to the podcast on Spotify, Apple Podcasts, Deezer, Amazon Music, among others, or directly via RSS feed.
