Development and Validation of a Robust and Interpretable Early Triaging Support System for Patients Hospitalized With COVID-19 : Predictive Algorithm Modeling and Interpretation Study

©Sangwon Baek, Yeon joo Jeong, Yun-Hyeon Kim, Jin Young Kim, Jin Hwan Kim, Eun Young Kim, Jae-Kwang Lim, Jungok Kim, Zero Kim, Kyunga Kim, Myung Jin Chung. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 11.01.2024..

BACKGROUND: Robust and accurate prediction of severity for patients with COVID-19 is crucial for patient triaging decisions. Many proposed models were prone to either high bias risk or low-to-moderate discrimination. Some also suffered from a lack of clinical interpretability and were developed based on early pandemic period data. Hence, there has been a compelling need for advancements in prediction models for better clinical applicability.

OBJECTIVE: The primary objective of this study was to develop and validate a machine learning-based Robust and Interpretable Early Triaging Support (RIETS) system that predicts severity progression (involving any of the following events: intensive care unit admission, in-hospital death, mechanical ventilation required, or extracorporeal membrane oxygenation required) within 15 days upon hospitalization based on routinely available clinical and laboratory biomarkers.

METHODS: We included data from 5945 hospitalized patients with COVID-19 from 19 hospitals in South Korea collected between January 2020 and August 2022. For model development and external validation, the whole data set was partitioned into 2 independent cohorts by stratified random cluster sampling according to hospital type (general and tertiary care) and geographical location (metropolitan and nonmetropolitan). Machine learning models were trained and internally validated through a cross-validation technique on the development cohort. They were externally validated using a bootstrapped sampling technique on the external validation cohort. The best-performing model was selected primarily based on the area under the receiver operating characteristic curve (AUROC), and its robustness was evaluated using bias risk assessment. For model interpretability, we used Shapley and patient clustering methods.

RESULTS: Our final model, RIETS, was developed based on a deep neural network of 11 clinical and laboratory biomarkers that are readily available within the first day of hospitalization. The features predictive of severity included lactate dehydrogenase, age, absolute lymphocyte count, dyspnea, respiratory rate, diabetes mellitus, c-reactive protein, absolute neutrophil count, platelet count, white blood cell count, and saturation of peripheral oxygen. RIETS demonstrated excellent discrimination (AUROC=0.937; 95% CI 0.935-0.938) with high calibration (integrated calibration index=0.041), satisfied all the criteria of low bias risk in a risk assessment tool, and provided detailed interpretations of model parameters and patient clusters. In addition, RIETS showed potential for transportability across variant periods with its sustainable prediction on Omicron cases (AUROC=0.903, 95% CI 0.897-0.910).

CONCLUSIONS: RIETS was developed and validated to assist early triaging by promptly predicting the severity of hospitalized patients with COVID-19. Its high performance with low bias risk ensures considerably reliable prediction. The use of a nationwide multicenter cohort in the model development and validation implicates generalizability. The use of routinely collected features may enable wide adaptability. Interpretations of model parameters and patients can promote clinical applicability. Together, we anticipate that RIETS will facilitate the patient triaging workflow and efficient resource allocation when incorporated into a routine clinical practice.

Medienart:	E-Artikel

Erscheinungsjahr:	2024
Erschienen:	2024

Enthalten in:	Zur Gesamtaufnahme - volume:26
Enthalten in:	Journal of medical Internet research - 26(2024) vom: 11. Jan., Seite e52134

Sprache:	Englisch

Beteiligte Personen:	Baek, Sangwon [VerfasserIn] Jeong, Yeon Joo [VerfasserIn] Kim, Yun-Hyeon [VerfasserIn] Kim, Jin Young [VerfasserIn] Kim, Jin Hwan [VerfasserIn] Kim, Eun Young [VerfasserIn] Lim, Jae-Kwang [VerfasserIn] Kim, Jungok [VerfasserIn] Kim, Zero [VerfasserIn] Kim, Kyunga [VerfasserIn] Chung, Myung Jin [VerfasserIn]

Links:	Volltext

Themen:

Biomarker Biomarkers COVID-19 Clustering Coronavirus Deep learning Early triaging Emergency Hospital admission Hospital admissions Hospitalization Hospitalizations Hospitalize Interpretability Journal Article Machine learning Multicenter Study Neural network Neural networks Omicron Predict Prediction Prediction model Predictive Prognosis Prognostic Prognostics SARS-CoV-2 SHAP Severity Shapley Triage Triaging Validation Study

Anmerkungen:	Date Completed 12.01.2024 Date Revised 28.01.2024 published: Electronic Citation Status MEDLINE

doi:	10.2196/52134

funding:
Förderinstitution / Projekttitel:

PPN (Katalog-ID):	NLM36697260X