Development and Validation of an Interpretable Conformal Predictor to Predict Sepsis Mortality Risk : Retrospective Cohort Study
©Meicheng Yang, Hui Chen, Wenhan Hu, Massimo Mischi, Caifeng Shan, Jianqing Li, Xi Long, Chengyu Liu. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 18.03.2024..
BACKGROUND: Early and reliable identification of patients with sepsis who are at high risk of mortality is important to improve clinical outcomes. However, 3 major barriers to artificial intelligence (AI) models, including the lack of interpretability, the difficulty in generalizability, and the risk of automation bias, hinder the widespread adoption of AI models for use in clinical practice.
OBJECTIVE: This study aimed to develop and validate (internally and externally) a conformal predictor of sepsis mortality risk in patients who are critically ill, leveraging AI-assisted prediction modeling. The proposed approach enables explaining the model output and assessing its confidence level.
METHODS: We retrospectively extracted data on adult patients with sepsis from a database collected in a teaching hospital at Beth Israel Deaconess Medical Center for model training and internal validation. A large multicenter critical care database from the Philips eICU Research Institute was used for external validation. A total of 103 clinical features were extracted from the first day after admission. We developed an AI model using gradient-boosting machines to predict the mortality risk of sepsis and used Mondrian conformal prediction to estimate the prediction uncertainty. The Shapley additive explanation method was used to explain the model.
RESULTS: A total of 16,746 (80%) patients from Beth Israel Deaconess Medical Center were used to train the model. When tested on the internal validation population of 4187 (20%) patients, the model achieved an area under the receiver operating characteristic curve of 0.858 (95% CI 0.845-0.871), which was reduced to 0.800 (95% CI 0.789-0.811) when externally validated on 10,362 patients from the Philips eICU database. At a specified confidence level of 90% for the internal validation cohort the percentage of error predictions (n=438) out of all predictions (n=4187) was 10.5%, with 1229 (29.4%) predictions requiring clinician review. In contrast, the AI model without conformal prediction made 1449 (34.6%) errors. When externally validated, more predictions (n=4004, 38.6%) were flagged for clinician review due to interdatabase heterogeneity. Nevertheless, the model still produced significantly lower error rates compared to the point predictions by AI (n=1221, 11.8% vs n=4540, 43.8%). The most important predictors identified in this predictive model were Acute Physiology Score III, age, urine output, vasopressors, and pulmonary infection. Clinically relevant risk factors contributing to a single patient were also examined to show how the risk arose.
CONCLUSIONS: By combining model explanation and conformal prediction, AI-based systems can be better translated into medical practice for clinical decision-making.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2024 |
|---|---|
| Erschienen: |
2024 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:26 |
|---|---|
| Enthalten in: |
Journal of medical Internet research - 26(2024) vom: 18. März, Seite e50369 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Yang, Meicheng [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
Clinical decision-making |
|---|
| Anmerkungen: |
Date Completed 19.03.2024 Date Revised 04.04.2024 published: Electronic Citation Status MEDLINE |
|---|
| doi: |
10.2196/50369 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM369877225 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM369877225 | ||
| 003 | DE-627 | ||
| 005 | 20240404235043.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 240318s2024 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.2196/50369 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1364.xml |
| 035 | |a (DE-627)NLM369877225 | ||
| 035 | |a (NLM)38498038 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Yang, Meicheng |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Development and Validation of an Interpretable Conformal Predictor to Predict Sepsis Mortality Risk |b Retrospective Cohort Study |
| 264 | 1 | |c 2024 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 19.03.2024 | ||
| 500 | |a Date Revised 04.04.2024 | ||
| 500 | |a published: Electronic | ||
| 500 | |a Citation Status MEDLINE | ||
| 520 | |a ©Meicheng Yang, Hui Chen, Wenhan Hu, Massimo Mischi, Caifeng Shan, Jianqing Li, Xi Long, Chengyu Liu. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 18.03.2024. | ||
| 520 | |a BACKGROUND: Early and reliable identification of patients with sepsis who are at high risk of mortality is important to improve clinical outcomes. However, 3 major barriers to artificial intelligence (AI) models, including the lack of interpretability, the difficulty in generalizability, and the risk of automation bias, hinder the widespread adoption of AI models for use in clinical practice | ||
| 520 | |a OBJECTIVE: This study aimed to develop and validate (internally and externally) a conformal predictor of sepsis mortality risk in patients who are critically ill, leveraging AI-assisted prediction modeling. The proposed approach enables explaining the model output and assessing its confidence level | ||
| 520 | |a METHODS: We retrospectively extracted data on adult patients with sepsis from a database collected in a teaching hospital at Beth Israel Deaconess Medical Center for model training and internal validation. A large multicenter critical care database from the Philips eICU Research Institute was used for external validation. A total of 103 clinical features were extracted from the first day after admission. We developed an AI model using gradient-boosting machines to predict the mortality risk of sepsis and used Mondrian conformal prediction to estimate the prediction uncertainty. The Shapley additive explanation method was used to explain the model | ||
| 520 | |a RESULTS: A total of 16,746 (80%) patients from Beth Israel Deaconess Medical Center were used to train the model. When tested on the internal validation population of 4187 (20%) patients, the model achieved an area under the receiver operating characteristic curve of 0.858 (95% CI 0.845-0.871), which was reduced to 0.800 (95% CI 0.789-0.811) when externally validated on 10,362 patients from the Philips eICU database. At a specified confidence level of 90% for the internal validation cohort the percentage of error predictions (n=438) out of all predictions (n=4187) was 10.5%, with 1229 (29.4%) predictions requiring clinician review. In contrast, the AI model without conformal prediction made 1449 (34.6%) errors. When externally validated, more predictions (n=4004, 38.6%) were flagged for clinician review due to interdatabase heterogeneity. Nevertheless, the model still produced significantly lower error rates compared to the point predictions by AI (n=1221, 11.8% vs n=4540, 43.8%). The most important predictors identified in this predictive model were Acute Physiology Score III, age, urine output, vasopressors, and pulmonary infection. Clinically relevant risk factors contributing to a single patient were also examined to show how the risk arose | ||
| 520 | |a CONCLUSIONS: By combining model explanation and conformal prediction, AI-based systems can be better translated into medical practice for clinical decision-making | ||
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a clinical decision-making | |
| 650 | 4 | |a conformal prediction | |
| 650 | 4 | |a critical care | |
| 650 | 4 | |a mortality prediction | |
| 650 | 4 | |a sepsis | |
| 700 | 1 | |a Chen, Hui |e verfasserin |4 aut | |
| 700 | 1 | |a Hu, Wenhan |e verfasserin |4 aut | |
| 700 | 1 | |a Mischi, Massimo |e verfasserin |4 aut | |
| 700 | 1 | |a Shan, Caifeng |e verfasserin |4 aut | |
| 700 | 1 | |a Li, Jianqing |e verfasserin |4 aut | |
| 700 | 1 | |a Long, Xi |e verfasserin |4 aut | |
| 700 | 1 | |a Liu, Chengyu |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t Journal of medical Internet research |d 1999 |g 26(2024) vom: 18. März, Seite e50369 |w (DE-627)NLM116127104 |x 1438-8871 |7 nnns |
| 773 | 1 | 8 | |g volume:26 |g year:2024 |g day:18 |g month:03 |g pages:e50369 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.2196/50369 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 26 |j 2024 |b 18 |c 03 |h e50369 | ||