Multi-Radiologist User Study for Artificial Intelligence-Guided Grading of COVID-19 Lung Disease Severity on Chest Radiographs
Copyright © 2021 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved..
RATIONALE AND OBJECTIVES: Radiographic findings of COVID-19 pneumonia can be used for patient risk stratification; however, radiologist reporting of disease severity is inconsistent on chest radiographs (CXRs). We aimed to see if an artificial intelligence (AI) system could help improve radiologist interrater agreement.
MATERIALS AND METHODS: We performed a retrospective multi-radiologist user study to evaluate the impact of an AI system, the PXS score model, on the grading of categorical COVID-19 lung disease severity on 154 chest radiographs into four ordinal grades (normal/minimal, mild, moderate, and severe). Four radiologists (two thoracic and two emergency radiologists) independently interpreted 154 CXRs from 154 unique patients with COVID-19 hospitalized at a large academic center, before and after using the AI system (median washout time interval was 16 days). Three different thoracic radiologists assessed the same 154 CXRs using an updated version of the AI system trained on more imaging data. Radiologist interrater agreement was evaluated using Cohen and Fleiss kappa where appropriate. The lung disease severity categories were associated with clinical outcomes using a previously published outcomes dataset using Fisher's exact test and Chi-square test for trend.
RESULTS: Use of the AI system improved radiologist interrater agreement (Fleiss κ = 0.40 to 0.66, before and after use of the system). The Fleiss κ for three radiologists using the updated AI system was 0.74. Severity categories were significantly associated with subsequent intubation or death within 3 days.
CONCLUSION: An AI system used at the time of CXR study interpretation can improve the interrater agreement of radiologists.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2021 |
|---|---|
| Erschienen: |
2021 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:28 |
|---|---|
| Enthalten in: |
Academic radiology - 28(2021), 4 vom: 20. Apr., Seite 572-576 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Li, Matthew D [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
Artificial intelligence |
|---|
| Anmerkungen: |
Date Completed 25.03.2021 Date Revised 10.11.2023 published: Print-Electronic Citation Status MEDLINE |
|---|
| doi: |
10.1016/j.acra.2021.01.016 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM320479161 |
|---|
| LEADER | 01000naa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM320479161 | ||
| 003 | DE-627 | ||
| 005 | 20231225173654.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231225s2021 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1016/j.acra.2021.01.016 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1068.xml |
| 035 | |a (DE-627)NLM320479161 | ||
| 035 | |a (NLM)33485773 | ||
| 035 | |a (PII)S1076-6332(21)00025-8 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Li, Matthew D |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Multi-Radiologist User Study for Artificial Intelligence-Guided Grading of COVID-19 Lung Disease Severity on Chest Radiographs |
| 264 | 1 | |c 2021 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 25.03.2021 | ||
| 500 | |a Date Revised 10.11.2023 | ||
| 500 | |a published: Print-Electronic | ||
| 500 | |a Citation Status MEDLINE | ||
| 520 | |a Copyright © 2021 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved. | ||
| 520 | |a RATIONALE AND OBJECTIVES: Radiographic findings of COVID-19 pneumonia can be used for patient risk stratification; however, radiologist reporting of disease severity is inconsistent on chest radiographs (CXRs). We aimed to see if an artificial intelligence (AI) system could help improve radiologist interrater agreement | ||
| 520 | |a MATERIALS AND METHODS: We performed a retrospective multi-radiologist user study to evaluate the impact of an AI system, the PXS score model, on the grading of categorical COVID-19 lung disease severity on 154 chest radiographs into four ordinal grades (normal/minimal, mild, moderate, and severe). Four radiologists (two thoracic and two emergency radiologists) independently interpreted 154 CXRs from 154 unique patients with COVID-19 hospitalized at a large academic center, before and after using the AI system (median washout time interval was 16 days). Three different thoracic radiologists assessed the same 154 CXRs using an updated version of the AI system trained on more imaging data. Radiologist interrater agreement was evaluated using Cohen and Fleiss kappa where appropriate. The lung disease severity categories were associated with clinical outcomes using a previously published outcomes dataset using Fisher's exact test and Chi-square test for trend | ||
| 520 | |a RESULTS: Use of the AI system improved radiologist interrater agreement (Fleiss κ = 0.40 to 0.66, before and after use of the system). The Fleiss κ for three radiologists using the updated AI system was 0.74. Severity categories were significantly associated with subsequent intubation or death within 3 days | ||
| 520 | |a CONCLUSION: An AI system used at the time of CXR study interpretation can improve the interrater agreement of radiologists | ||
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a Research Support, Non-U.S. Gov't | |
| 650 | 4 | |a Artificial intelligence | |
| 650 | 4 | |a COVID-19 | |
| 650 | 4 | |a Chest radiograph | |
| 650 | 4 | |a Computer-assisted diagnosis | |
| 700 | 1 | |a Little, Brent P |e verfasserin |4 aut | |
| 700 | 1 | |a Alkasab, Tarik K |e verfasserin |4 aut | |
| 700 | 1 | |a Mendoza, Dexter P |e verfasserin |4 aut | |
| 700 | 1 | |a Succi, Marc D |e verfasserin |4 aut | |
| 700 | 1 | |a Shepard, Jo-Anne O |e verfasserin |4 aut | |
| 700 | 1 | |a Lev, Michael H |e verfasserin |4 aut | |
| 700 | 1 | |a Kalpathy-Cramer, Jayashree |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t Academic radiology |d 1995 |g 28(2021), 4 vom: 20. Apr., Seite 572-576 |w (DE-627)NLM087676818 |x 1878-4046 |7 nnns |
| 773 | 1 | 8 | |g volume:28 |g year:2021 |g number:4 |g day:20 |g month:04 |g pages:572-576 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1016/j.acra.2021.01.016 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 28 |j 2021 |e 4 |b 20 |c 04 |h 572-576 | ||