Machine learning in clinical practice : Evaluation of an artificial intelligence tool after implementation
© 2023 The Authors. Emergency Medicine Australasia published by John Wiley & Sons Australia, Ltd on behalf of Australasian College for Emergency Medicine..
OBJECTIVE: Artificial intelligence (AI) has gradually found its way into healthcare, and its future integration into clinical practice is inevitable. In the present study, we evaluate the accuracy of a novel AI algorithm designed to predict admission based on a triage note after clinical implementation. This is the first of such studies to investigate real-time AI performance in the emergency setting.
METHODS: The novel AI algorithm that predicts admission using a triage note was translated into clinical practice and integrated within St Vincent's Hospital Melbourne's electronic emergency patient management system. The data were collected from 1 January 2021 to 17 August 2022 to evaluate the diagnostic accuracy of the AI system after implementation.
RESULTS: A total of 77 125 ED presentations were included. The live AI algorithm has a sensitivity of 73.1% (95% confidence interval 72.5-73.8), specificity of 74.3% (73.9-74.7), positive predictive value of 50% (49.6-50.4) and negative predictive value of 88.7% (88.5-89) with a total accuracy of 74% (73.7-74.3). The accuracy of the system was at the lowest for admission to psychiatric units (34%) and at the highest for gastroenterology and medical admission (84% and 80%, respectively).
CONCLUSION: Our study showed the diagnostic evaluation of a real-time AI clinical decision-support tool became less accurate than the original. Although real-time sensitivity and specificity of the AI tool was still acceptable as a decision-support tool in the ED, we propose that continuous training and evaluation of AI-enabled clinical support tools in healthcare are conducted to ensure consistent accuracy and performance to prevent inadvertent consequences.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2024 |
|---|---|
| Erschienen: |
2024 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:36 |
|---|---|
| Enthalten in: |
Emergency medicine Australasia : EMA - 36(2024), 1 vom: 28. Jan., Seite 118-124 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Akhlaghi, Hamed [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
Artificial intelligence |
|---|
| Anmerkungen: |
Date Completed 18.01.2024 Date Revised 18.01.2024 published: Print-Electronic Citation Status MEDLINE |
|---|
| doi: |
10.1111/1742-6723.14325 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM362672822 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM362672822 | ||
| 003 | DE-627 | ||
| 005 | 20240118231919.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231226s2024 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1111/1742-6723.14325 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1263.xml |
| 035 | |a (DE-627)NLM362672822 | ||
| 035 | |a (NLM)37771067 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Akhlaghi, Hamed |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Machine learning in clinical practice |b Evaluation of an artificial intelligence tool after implementation |
| 264 | 1 | |c 2024 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 18.01.2024 | ||
| 500 | |a Date Revised 18.01.2024 | ||
| 500 | |a published: Print-Electronic | ||
| 500 | |a Citation Status MEDLINE | ||
| 520 | |a © 2023 The Authors. Emergency Medicine Australasia published by John Wiley & Sons Australia, Ltd on behalf of Australasian College for Emergency Medicine. | ||
| 520 | |a OBJECTIVE: Artificial intelligence (AI) has gradually found its way into healthcare, and its future integration into clinical practice is inevitable. In the present study, we evaluate the accuracy of a novel AI algorithm designed to predict admission based on a triage note after clinical implementation. This is the first of such studies to investigate real-time AI performance in the emergency setting | ||
| 520 | |a METHODS: The novel AI algorithm that predicts admission using a triage note was translated into clinical practice and integrated within St Vincent's Hospital Melbourne's electronic emergency patient management system. The data were collected from 1 January 2021 to 17 August 2022 to evaluate the diagnostic accuracy of the AI system after implementation | ||
| 520 | |a RESULTS: A total of 77 125 ED presentations were included. The live AI algorithm has a sensitivity of 73.1% (95% confidence interval 72.5-73.8), specificity of 74.3% (73.9-74.7), positive predictive value of 50% (49.6-50.4) and negative predictive value of 88.7% (88.5-89) with a total accuracy of 74% (73.7-74.3). The accuracy of the system was at the lowest for admission to psychiatric units (34%) and at the highest for gastroenterology and medical admission (84% and 80%, respectively) | ||
| 520 | |a CONCLUSION: Our study showed the diagnostic evaluation of a real-time AI clinical decision-support tool became less accurate than the original. Although real-time sensitivity and specificity of the AI tool was still acceptable as a decision-support tool in the ED, we propose that continuous training and evaluation of AI-enabled clinical support tools in healthcare are conducted to ensure consistent accuracy and performance to prevent inadvertent consequences | ||
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a artificial intelligence | |
| 650 | 4 | |a emergency department | |
| 650 | 4 | |a machine learning | |
| 650 | 4 | |a research translation | |
| 650 | 4 | |a triage note | |
| 700 | 1 | |a Freeman, Sam |e verfasserin |4 aut | |
| 700 | 1 | |a Vari, Cynthia |e verfasserin |4 aut | |
| 700 | 1 | |a McKenna, Bede |e verfasserin |4 aut | |
| 700 | 1 | |a Braitberg, George |e verfasserin |4 aut | |
| 700 | 1 | |a Karro, Jonathan |e verfasserin |4 aut | |
| 700 | 1 | |a Tahayori, Bahman |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t Emergency medicine Australasia : EMA |d 2004 |g 36(2024), 1 vom: 28. Jan., Seite 118-124 |w (DE-627)NLM149139780 |x 1742-6723 |7 nnns |
| 773 | 1 | 8 | |g volume:36 |g year:2024 |g number:1 |g day:28 |g month:01 |g pages:118-124 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1111/1742-6723.14325 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 36 |j 2024 |e 1 |b 28 |c 01 |h 118-124 | ||