When the machine is wrong. Characteristics of true and false predictions of Out-of-Hospital Cardiac arrests in emergency calls using a machine-learning model
Copyright © 2023 The Author(s). Published by Elsevier B.V. All rights reserved..
BACKGROUND: A machine-learning model trained to recognize emergency calls regarding Out-of-Hospital Cardiac Arrest (OHCA) was tested in clinical practice at Copenhagen Emergency Medical Services (EMS) from September 2018 to December 2019. We aimed to investigate emergency call characteristics where the machine-learning model failed to recognize OHCA or misinterpreted a call as being OHCA.
METHODS: All emergency calls were linked to the dispatch database and verified OHCAs were identified by linkage to the Danish Cardiac Arrest Registry. Calls with either false negative or false positive predictions of OHCA were evaluated by trained auditors. Descriptive analyses were performed with absolute numbers and percentages reported.
RESULTS: The machine-learning model processed 169,236 calls to Copenhagen EMS and suspected 5,811 (3.4%) of the calls as OHCA, resulting in 84.5% sensitivity and 97.1% specificity. Among OHCAs not recognised by machine-learning model, a condition completely different from OHCA was presented by caller in 31% of the cases. In 28% of unrecognised calls, patient was reported breathing normally, and language barriers were identified in 23% of the cases. Among falsely suspected OHCA, the patient was reported unconscious in 28% of the cases, and in 13% of the false positive cases the machine-learning model interpreted calls regarding dead patients with irreversible signs of death as OHCA.
CONCLUSION: Continuous optimization of the language model is needed to improve the prediction of OHCA and thereby improve sensitivity and specificity of the machine-learning model on recognising OHCA in emergency telephone calls.
Medienart: |
E-Artikel |
---|
Erscheinungsjahr: |
2023 |
---|---|
Erschienen: |
2023 |
Enthalten in: |
Zur Gesamtaufnahme - volume:183 |
---|---|
Enthalten in: |
Resuscitation - 183(2023) vom: 15. Feb., Seite 109689 |
Sprache: |
Englisch |
---|
Beteiligte Personen: |
Nikolaj Blomberg, Stig [VerfasserIn] |
---|
Links: |
---|
Themen: |
AI |
---|
Anmerkungen: |
Date Completed 07.02.2023 Date Revised 07.03.2023 published: Print-Electronic Citation Status MEDLINE |
---|
doi: |
10.1016/j.resuscitation.2023.109689 |
---|
funding: |
|
---|---|
Förderinstitution / Projekttitel: |
|
PPN (Katalog-ID): |
NLM351457097 |
---|
LEADER | 01000naa a22002652 4500 | ||
---|---|---|---|
001 | NLM351457097 | ||
003 | DE-627 | ||
005 | 20231226051419.0 | ||
007 | cr uuu---uuuuu | ||
008 | 231226s2023 xx |||||o 00| ||eng c | ||
024 | 7 | |a 10.1016/j.resuscitation.2023.109689 |2 doi | |
028 | 5 | 2 | |a pubmed24n1171.xml |
035 | |a (DE-627)NLM351457097 | ||
035 | |a (NLM)36634755 | ||
035 | |a (PII)S0300-9572(23)00002-3 | ||
040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
041 | |a eng | ||
100 | 1 | |a Nikolaj Blomberg, Stig |e verfasserin |4 aut | |
245 | 1 | 0 | |a When the machine is wrong. Characteristics of true and false predictions of Out-of-Hospital Cardiac arrests in emergency calls using a machine-learning model |
264 | 1 | |c 2023 | |
336 | |a Text |b txt |2 rdacontent | ||
337 | |a ƒaComputermedien |b c |2 rdamedia | ||
338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
500 | |a Date Completed 07.02.2023 | ||
500 | |a Date Revised 07.03.2023 | ||
500 | |a published: Print-Electronic | ||
500 | |a Citation Status MEDLINE | ||
520 | |a Copyright © 2023 The Author(s). Published by Elsevier B.V. All rights reserved. | ||
520 | |a BACKGROUND: A machine-learning model trained to recognize emergency calls regarding Out-of-Hospital Cardiac Arrest (OHCA) was tested in clinical practice at Copenhagen Emergency Medical Services (EMS) from September 2018 to December 2019. We aimed to investigate emergency call characteristics where the machine-learning model failed to recognize OHCA or misinterpreted a call as being OHCA | ||
520 | |a METHODS: All emergency calls were linked to the dispatch database and verified OHCAs were identified by linkage to the Danish Cardiac Arrest Registry. Calls with either false negative or false positive predictions of OHCA were evaluated by trained auditors. Descriptive analyses were performed with absolute numbers and percentages reported | ||
520 | |a RESULTS: The machine-learning model processed 169,236 calls to Copenhagen EMS and suspected 5,811 (3.4%) of the calls as OHCA, resulting in 84.5% sensitivity and 97.1% specificity. Among OHCAs not recognised by machine-learning model, a condition completely different from OHCA was presented by caller in 31% of the cases. In 28% of unrecognised calls, patient was reported breathing normally, and language barriers were identified in 23% of the cases. Among falsely suspected OHCA, the patient was reported unconscious in 28% of the cases, and in 13% of the false positive cases the machine-learning model interpreted calls regarding dead patients with irreversible signs of death as OHCA | ||
520 | |a CONCLUSION: Continuous optimization of the language model is needed to improve the prediction of OHCA and thereby improve sensitivity and specificity of the machine-learning model on recognising OHCA in emergency telephone calls | ||
650 | 4 | |a Journal Article | |
650 | 4 | |a Research Support, Non-U.S. Gov't | |
650 | 4 | |a AI | |
650 | 4 | |a Dispatch | |
650 | 4 | |a Emergency Medicine | |
650 | 4 | |a Machine-learning | |
650 | 4 | |a OHCA | |
650 | 4 | |a Out-of-Hospital Cardiac Arrest, cardiology | |
700 | 1 | |a Jensen, Theo W |e verfasserin |4 aut | |
700 | 1 | |a Porsborg Andersen, Mikkel |e verfasserin |4 aut | |
700 | 1 | |a Folke, Fredrik |e verfasserin |4 aut | |
700 | 1 | |a Kjær Ersbøll, Annette |e verfasserin |4 aut | |
700 | 1 | |a Torp-Petersen, Christian |e verfasserin |4 aut | |
700 | 1 | |a Lippert, Freddy |e verfasserin |4 aut | |
700 | 1 | |a Collatz Christensen, Helle |e verfasserin |4 aut | |
773 | 0 | 8 | |i Enthalten in |t Resuscitation |d 1972 |g 183(2023) vom: 15. Feb., Seite 109689 |w (DE-627)NLM000010049 |x 1873-1570 |7 nnns |
773 | 1 | 8 | |g volume:183 |g year:2023 |g day:15 |g month:02 |g pages:109689 |
856 | 4 | 0 | |u http://dx.doi.org/10.1016/j.resuscitation.2023.109689 |3 Volltext |
912 | |a GBV_USEFLAG_A | ||
912 | |a GBV_NLM | ||
951 | |a AR | ||
952 | |d 183 |j 2023 |b 15 |c 02 |h 109689 |