A Multicenter, Scan-Rescan, Human and Machine Learning CMR Study to Test Generalizability and Precision in Imaging Biomarker Analysis
BACKGROUND: Automated analysis of cardiac structure and function using machine learning (ML) has great potential, but is currently hindered by poor generalizability. Comparison is traditionally against clinicians as a reference, ignoring inherent human inter- and intraobserver error, and ensuring that ML cannot demonstrate superiority. Measuring precision (scan:rescan reproducibility) addresses this. We compared precision of ML and humans using a multicenter, multi-disease, scan:rescan cardiovascular magnetic resonance data set.
METHODS: One hundred ten patients (5 disease categories, 5 institutions, 2 scanner manufacturers, and 2 field strengths) underwent scan:rescan cardiovascular magnetic resonance (96% within one week). After identification of the most precise human technique, left ventricular chamber volumes, mass, and ejection fraction were measured by an expert, a trained junior clinician, and a fully automated convolutional neural network trained on 599 independent multicenter disease cases. Scan:rescan coefficient of variation and 1000 bootstrapped 95% CIs were calculated and compared using mixed linear effects models.
RESULTS: Clinicians can be confident in detecting a 9% change in left ventricular ejection fraction, with greater than half of coefficient of variation attributable to intraobserver variation. Expert, trained junior, and automated scan:rescan precision were similar (for left ventricular ejection fraction, coefficient of variation 6.1 [5.2%-7.1%], P=0.2581; 8.3 [5.6%-10.3%], P=0.3653; 8.8 [6.1%-11.1%], P=0.8620). Automated analysis was 186× faster than humans (0.07 versus 13 minutes).
CONCLUSIONS: Automated ML analysis is faster with similar precision to the most precise human techniques, even when challenged with real-world scan:rescan data. Assessment of multicenter, multi-vendor, multi-field strength scan:rescan data (available at www.thevolumesresource.com) permits a generalizable assessment of ML precision and may facilitate direct translation of ML to clinical practice.
| Errataetall: |
CommentIn: Circ Cardiovasc Imaging. 2019 Oct;12(10):e009759. - PMID 31547690 |
|---|---|
| Medienart: |
E-Artikel |
| Erscheinungsjahr: |
2019 |
|---|---|
| Erschienen: |
2019 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:12 |
|---|---|
| Enthalten in: |
Circulation. Cardiovascular imaging - 12(2019), 10 vom: 28. Okt., Seite e009214 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Bhuva, Anish [VerfasserIn] |
|---|
| Links: |
|---|
| Anmerkungen: |
Date Completed 08.06.2020 Date Revised 06.03.2024 published: Print-Electronic CommentIn: Circ Cardiovasc Imaging. 2019 Oct;12(10):e009759. - PMID 31547690 Citation Status MEDLINE |
|---|
| doi: |
10.1161/CIRCIMAGING.119.009214 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM301556296 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM301556296 | ||
| 003 | DE-627 | ||
| 005 | 20240306231922.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231225s2019 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1161/CIRCIMAGING.119.009214 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1318.xml |
| 035 | |a (DE-627)NLM301556296 | ||
| 035 | |a (NLM)31547689 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Bhuva, Anish |e verfasserin |4 aut | |
| 245 | 1 | 2 | |a A Multicenter, Scan-Rescan, Human and Machine Learning CMR Study to Test Generalizability and Precision in Imaging Biomarker Analysis |
| 264 | 1 | |c 2019 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 08.06.2020 | ||
| 500 | |a Date Revised 06.03.2024 | ||
| 500 | |a published: Print-Electronic | ||
| 500 | |a CommentIn: Circ Cardiovasc Imaging. 2019 Oct;12(10):e009759. - PMID 31547690 | ||
| 500 | |a Citation Status MEDLINE | ||
| 520 | |a BACKGROUND: Automated analysis of cardiac structure and function using machine learning (ML) has great potential, but is currently hindered by poor generalizability. Comparison is traditionally against clinicians as a reference, ignoring inherent human inter- and intraobserver error, and ensuring that ML cannot demonstrate superiority. Measuring precision (scan:rescan reproducibility) addresses this. We compared precision of ML and humans using a multicenter, multi-disease, scan:rescan cardiovascular magnetic resonance data set | ||
| 520 | |a METHODS: One hundred ten patients (5 disease categories, 5 institutions, 2 scanner manufacturers, and 2 field strengths) underwent scan:rescan cardiovascular magnetic resonance (96% within one week). After identification of the most precise human technique, left ventricular chamber volumes, mass, and ejection fraction were measured by an expert, a trained junior clinician, and a fully automated convolutional neural network trained on 599 independent multicenter disease cases. Scan:rescan coefficient of variation and 1000 bootstrapped 95% CIs were calculated and compared using mixed linear effects models | ||
| 520 | |a RESULTS: Clinicians can be confident in detecting a 9% change in left ventricular ejection fraction, with greater than half of coefficient of variation attributable to intraobserver variation. Expert, trained junior, and automated scan:rescan precision were similar (for left ventricular ejection fraction, coefficient of variation 6.1 [5.2%-7.1%], P=0.2581; 8.3 [5.6%-10.3%], P=0.3653; 8.8 [6.1%-11.1%], P=0.8620). Automated analysis was 186× faster than humans (0.07 versus 13 minutes) | ||
| 520 | |a CONCLUSIONS: Automated ML analysis is faster with similar precision to the most precise human techniques, even when challenged with real-world scan:rescan data. Assessment of multicenter, multi-vendor, multi-field strength scan:rescan data (available at www.thevolumesresource.com) permits a generalizable assessment of ML precision and may facilitate direct translation of ML to clinical practice | ||
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a Multicenter Study | |
| 650 | 4 | |a Research Support, Non-U.S. Gov't | |
| 650 | 4 | |a artificial intelligence | |
| 650 | 4 | |a image processing | |
| 650 | 4 | |a left ventricular remodeling | |
| 650 | 4 | |a magnetic resonance imaging, cine | |
| 650 | 4 | |a ventricular function | |
| 650 | 7 | |a Biomarkers |2 NLM | |
| 700 | 1 | |a Bai, Wenjia |e verfasserin |4 aut | |
| 700 | 1 | |a Lau, Clement |e verfasserin |4 aut | |
| 700 | 1 | |a Davies, Rhodri |e verfasserin |4 aut | |
| 700 | 1 | |a Ye, Yang |e verfasserin |4 aut | |
| 700 | 1 | |a Bulluck, Heeraj |e verfasserin |4 aut | |
| 700 | 1 | |a McAlindon, Elisa |e verfasserin |4 aut | |
| 700 | 1 | |a Culotta, Veronica |e verfasserin |4 aut | |
| 700 | 1 | |a Swoboda, Peter |e verfasserin |4 aut | |
| 700 | 1 | |a Captur, Gabriella |e verfasserin |4 aut | |
| 700 | 1 | |a Treibel, Thomas |e verfasserin |4 aut | |
| 700 | 1 | |a Augusto, Joao |e verfasserin |4 aut | |
| 700 | 1 | |a Knott, Kristopher |e verfasserin |4 aut | |
| 700 | 1 | |a Seraphim, Andreas |e verfasserin |4 aut | |
| 700 | 1 | |a Cole, Graham |e verfasserin |4 aut | |
| 700 | 1 | |a Petersen, Steffen |e verfasserin |4 aut | |
| 700 | 1 | |a Edwards, Nicola |e verfasserin |4 aut | |
| 700 | 1 | |a Greenwood, John |e verfasserin |4 aut | |
| 700 | 1 | |a Bucciarelli-Ducci, Chiara |e verfasserin |4 aut | |
| 700 | 1 | |a Hughes, Alun |e verfasserin |4 aut | |
| 700 | 1 | |a Rueckert, Daniel |e verfasserin |4 aut | |
| 700 | 1 | |a Moon, James |e verfasserin |4 aut | |
| 700 | 1 | |a Manisty, Charlotte |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t Circulation. Cardiovascular imaging |d 2008 |g 12(2019), 10 vom: 28. Okt., Seite e009214 |w (DE-627)NLM191867586 |x 1942-0080 |7 nnns |
| 773 | 1 | 8 | |g volume:12 |g year:2019 |g number:10 |g day:28 |g month:10 |g pages:e009214 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1161/CIRCIMAGING.119.009214 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 12 |j 2019 |e 10 |b 28 |c 10 |h e009214 | ||