ChatGPT in medical school : how successful is AI in progress testing?
BACKGROUND: As generative artificial intelligence (AI), ChatGPT provides easy access to a wide range of information, including factual knowledge in the field of medicine. Given that knowledge acquisition is a basic determinant of physicians' performance, teaching and testing different levels of medical knowledge is a central task of medical schools. To measure the factual knowledge level of the ChatGPT responses, we compared the performance of ChatGPT with that of medical students in a progress test.
METHODS: A total of 400 multiple-choice questions (MCQs) from the progress test in German-speaking countries were entered into ChatGPT's user interface to obtain the percentage of correctly answered questions. We calculated the correlations of the correctness of ChatGPT responses with behavior in terms of response time, word count, and difficulty of a progress test question.
RESULTS: Of the 395 responses evaluated, 65.5% of the progress test questions answered by ChatGPT were correct. On average, ChatGPT required 22.8 s (SD 17.5) for a complete response, containing 36.2 (SD 28.1) words. There was no correlation between the time used and word count with the accuracy of the ChatGPT response (correlation coefficient for time rho = -0.08, 95% CI [-0.18, 0.02], t(393) = -1.55, p = 0.121; for word count rho = -0.03, 95% CI [-0.13, 0.07], t(393) = -0.54, p = 0.592). There was a significant correlation between the difficulty index of the MCQs and the accuracy of the ChatGPT response (correlation coefficient for difficulty: rho = 0.16, 95% CI [0.06, 0.25], t(393) = 3.19, p = 0.002).
CONCLUSION: ChatGPT was able to correctly answer two-thirds of all MCQs at the German state licensing exam level in Progress Test Medicine and outperformed almost all medical students in years 1-3. The ChatGPT answers can be compared with the performance of medical students in the second half of their studies.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2023 |
|---|---|
| Erschienen: |
2023 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:28 |
|---|---|
| Enthalten in: |
Medical education online - 28(2023), 1 vom: 12. Dez., Seite 2220920 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Friederichs, Hendrik [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
Artificial intelligence |
|---|
| Anmerkungen: |
Date Completed 29.09.2023 Date Revised 29.09.2023 published: Print Citation Status MEDLINE |
|---|
| doi: |
10.1080/10872981.2023.2220920 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM358092140 |
|---|
| LEADER | 01000naa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM358092140 | ||
| 003 | DE-627 | ||
| 005 | 20231226074015.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231226s2023 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1080/10872981.2023.2220920 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1193.xml |
| 035 | |a (DE-627)NLM358092140 | ||
| 035 | |a (NLM)37307503 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Friederichs, Hendrik |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a ChatGPT in medical school |b how successful is AI in progress testing? |
| 264 | 1 | |c 2023 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 29.09.2023 | ||
| 500 | |a Date Revised 29.09.2023 | ||
| 500 | |a published: Print | ||
| 500 | |a Citation Status MEDLINE | ||
| 520 | |a BACKGROUND: As generative artificial intelligence (AI), ChatGPT provides easy access to a wide range of information, including factual knowledge in the field of medicine. Given that knowledge acquisition is a basic determinant of physicians' performance, teaching and testing different levels of medical knowledge is a central task of medical schools. To measure the factual knowledge level of the ChatGPT responses, we compared the performance of ChatGPT with that of medical students in a progress test | ||
| 520 | |a METHODS: A total of 400 multiple-choice questions (MCQs) from the progress test in German-speaking countries were entered into ChatGPT's user interface to obtain the percentage of correctly answered questions. We calculated the correlations of the correctness of ChatGPT responses with behavior in terms of response time, word count, and difficulty of a progress test question | ||
| 520 | |a RESULTS: Of the 395 responses evaluated, 65.5% of the progress test questions answered by ChatGPT were correct. On average, ChatGPT required 22.8 s (SD 17.5) for a complete response, containing 36.2 (SD 28.1) words. There was no correlation between the time used and word count with the accuracy of the ChatGPT response (correlation coefficient for time rho = -0.08, 95% CI [-0.18, 0.02], t(393) = -1.55, p = 0.121; for word count rho = -0.03, 95% CI [-0.13, 0.07], t(393) = -0.54, p = 0.592). There was a significant correlation between the difficulty index of the MCQs and the accuracy of the ChatGPT response (correlation coefficient for difficulty: rho = 0.16, 95% CI [0.06, 0.25], t(393) = 3.19, p = 0.002) | ||
| 520 | |a CONCLUSION: ChatGPT was able to correctly answer two-thirds of all MCQs at the German state licensing exam level in Progress Test Medicine and outperformed almost all medical students in years 1-3. The ChatGPT answers can be compared with the performance of medical students in the second half of their studies | ||
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a Medical education | |
| 650 | 4 | |a artificial intelligence | |
| 650 | 4 | |a learning | |
| 650 | 4 | |a machine learning | |
| 650 | 4 | |a progress test | |
| 700 | 1 | |a Friederichs, Wolf Jonas |e verfasserin |4 aut | |
| 700 | 1 | |a März, Maren |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t Medical education online |d 1998 |g 28(2023), 1 vom: 12. Dez., Seite 2220920 |w (DE-627)NLM097445851 |x 1087-2981 |7 nnns |
| 773 | 1 | 8 | |g volume:28 |g year:2023 |g number:1 |g day:12 |g month:12 |g pages:2220920 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1080/10872981.2023.2220920 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 28 |j 2023 |e 1 |b 12 |c 12 |h 2220920 | ||