Physician Assessment of ChatGPT and Bing Answers to American Cancer Society's Questions to Ask About Your Cancer
Copyright © 2023 Wolters Kluwer Health, Inc. All rights reserved..
OBJECTIVES: Artificial intelligence (AI) chatbots are a new, publicly available tool for patients to access health care-related information with unknown reliability related to cancer-related questions. This study assesses the quality of responses to common questions for patients with cancer.
METHODS: From February to March 2023, we queried chat generative pretrained transformer (ChatGPT) from OpenAI and Bing AI from Microsoft questions from the American Cancer Society's recommended "Questions to Ask About Your Cancer" customized for all stages of breast, colon, lung, and prostate cancer. Questions were, in addition, grouped by type (prognosis, treatment, or miscellaneous). The quality of AI chatbot responses was assessed by an expert panel using the validated DISCERN criteria.
RESULTS: Of the 117 questions presented to ChatGPT and Bing, the average score for all questions were 3.9 and 3.2, respectively ( P < 0.001) and the overall DISCERN scores were 4.1 and 4.4, respectively. By disease site, the average score for ChatGPT and Bing, respectively, were 3.9 and 3.6 for prostate cancer ( P = 0.02), 3.7 and 3.3 for lung cancer ( P < 0.001), 4.1 and 2.9 for breast cancer ( P < 0.001), and 3.8 and 3.0 for colorectal cancer ( P < 0.001). By type of question, the average score for ChatGPT and Bing, respectively, were 3.6 and 3.4 for prognostic questions ( P = 0.12), 3.9 and 3.1 for treatment questions ( P < 0.001), and 4.2 and 3.3 for miscellaneous questions ( P = 0.001). For 3 responses (3%) by ChatGPT and 18 responses (15%) by Bing, at least one panelist rated them as having serious or extensive shortcomings.
CONCLUSIONS: AI chatbots provide multiple opportunities for innovating health care. This analysis suggests a critical need, particularly around cancer prognostication, for continual refinement to limit misleading counseling, confusion, and emotional distress to patients and families.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2024 |
|---|---|
| Erschienen: |
2024 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:47 |
|---|---|
| Enthalten in: |
American journal of clinical oncology - 47(2024), 1 vom: 01. Jan., Seite 17-21 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Janopaul-Naylor, James R [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
|---|
| Anmerkungen: |
Date Completed 28.12.2023 Date Revised 28.02.2024 published: Print-Electronic Citation Status MEDLINE |
|---|
| doi: |
10.1097/COC.0000000000001050 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM36316751X |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM36316751X | ||
| 003 | DE-627 | ||
| 005 | 20240229170327.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231226s2024 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1097/COC.0000000000001050 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1310.xml |
| 035 | |a (DE-627)NLM36316751X | ||
| 035 | |a (NLM)37823708 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Janopaul-Naylor, James R |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Physician Assessment of ChatGPT and Bing Answers to American Cancer Society's Questions to Ask About Your Cancer |
| 264 | 1 | |c 2024 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 28.12.2023 | ||
| 500 | |a Date Revised 28.02.2024 | ||
| 500 | |a published: Print-Electronic | ||
| 500 | |a Citation Status MEDLINE | ||
| 520 | |a Copyright © 2023 Wolters Kluwer Health, Inc. All rights reserved. | ||
| 520 | |a OBJECTIVES: Artificial intelligence (AI) chatbots are a new, publicly available tool for patients to access health care-related information with unknown reliability related to cancer-related questions. This study assesses the quality of responses to common questions for patients with cancer | ||
| 520 | |a METHODS: From February to March 2023, we queried chat generative pretrained transformer (ChatGPT) from OpenAI and Bing AI from Microsoft questions from the American Cancer Society's recommended "Questions to Ask About Your Cancer" customized for all stages of breast, colon, lung, and prostate cancer. Questions were, in addition, grouped by type (prognosis, treatment, or miscellaneous). The quality of AI chatbot responses was assessed by an expert panel using the validated DISCERN criteria | ||
| 520 | |a RESULTS: Of the 117 questions presented to ChatGPT and Bing, the average score for all questions were 3.9 and 3.2, respectively ( P < 0.001) and the overall DISCERN scores were 4.1 and 4.4, respectively. By disease site, the average score for ChatGPT and Bing, respectively, were 3.9 and 3.6 for prostate cancer ( P = 0.02), 3.7 and 3.3 for lung cancer ( P < 0.001), 4.1 and 2.9 for breast cancer ( P < 0.001), and 3.8 and 3.0 for colorectal cancer ( P < 0.001). By type of question, the average score for ChatGPT and Bing, respectively, were 3.6 and 3.4 for prognostic questions ( P = 0.12), 3.9 and 3.1 for treatment questions ( P < 0.001), and 4.2 and 3.3 for miscellaneous questions ( P = 0.001). For 3 responses (3%) by ChatGPT and 18 responses (15%) by Bing, at least one panelist rated them as having serious or extensive shortcomings | ||
| 520 | |a CONCLUSIONS: AI chatbots provide multiple opportunities for innovating health care. This analysis suggests a critical need, particularly around cancer prognostication, for continual refinement to limit misleading counseling, confusion, and emotional distress to patients and families | ||
| 650 | 4 | |a Journal Article | |
| 700 | 1 | |a Koo, Andee |e verfasserin |4 aut | |
| 700 | 1 | |a Qian, David C |e verfasserin |4 aut | |
| 700 | 1 | |a McCall, Neal S |e verfasserin |4 aut | |
| 700 | 1 | |a Liu, Yuan |e verfasserin |4 aut | |
| 700 | 1 | |a Patel, Sagar A |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t American journal of clinical oncology |d 1988 |g 47(2024), 1 vom: 01. Jan., Seite 17-21 |w (DE-627)NLM012605603 |x 1537-453X |7 nnns |
| 773 | 1 | 8 | |g volume:47 |g year:2024 |g number:1 |g day:01 |g month:01 |g pages:17-21 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1097/COC.0000000000001050 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 47 |j 2024 |e 1 |b 01 |c 01 |h 17-21 | ||