Evaluation of crowdsourced mortality prediction models as a framework for assessing artificial intelligence in medicine
© The Author(s) 2023. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissionsoup.com..
OBJECTIVE: Applications of machine learning in healthcare are of high interest and have the potential to improve patient care. Yet, the real-world accuracy of these models in clinical practice and on different patient subpopulations remains unclear. To address these important questions, we hosted a community challenge to evaluate methods that predict healthcare outcomes. We focused on the prediction of all-cause mortality as the community challenge question.
MATERIALS AND METHODS: Using a Model-to-Data framework, 345 registered participants, coalescing into 25 independent teams, spread over 3 continents and 10 countries, generated 25 accurate models all trained on a dataset of over 1.1 million patients and evaluated on patients prospectively collected over a 1-year observation of a large health system.
RESULTS: The top performing team achieved a final area under the receiver operator curve of 0.947 (95% CI, 0.942-0.951) and an area under the precision-recall curve of 0.487 (95% CI, 0.458-0.499) on a prospectively collected patient cohort.
DISCUSSION: Post hoc analysis after the challenge revealed that models differ in accuracy on subpopulations, delineated by race or gender, even when they are trained on the same data.
CONCLUSION: This is the largest community challenge focused on the evaluation of state-of-the-art machine learning methods in a healthcare system performed to date, revealing both opportunities and pitfalls of clinical AI.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2023 |
|---|---|
| Erschienen: |
2023 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:31 |
|---|---|
| Enthalten in: |
Journal of the American Medical Informatics Association : JAMIA - 31(2023), 1 vom: 22. Dez., Seite 35-44 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Bergquist, Timothy [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
Evaluation |
|---|
| Anmerkungen: |
Date Completed 25.12.2023 Date Revised 14.03.2024 published: Print Citation Status MEDLINE |
|---|
| doi: |
10.1093/jamia/ocad159 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM361027311 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM361027311 | ||
| 003 | DE-627 | ||
| 005 | 20240314234513.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231226s2023 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1093/jamia/ocad159 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1328.xml |
| 035 | |a (DE-627)NLM361027311 | ||
| 035 | |a (NLM)37604111 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Bergquist, Timothy |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Evaluation of crowdsourced mortality prediction models as a framework for assessing artificial intelligence in medicine |
| 264 | 1 | |c 2023 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 25.12.2023 | ||
| 500 | |a Date Revised 14.03.2024 | ||
| 500 | |a published: Print | ||
| 500 | |a Citation Status MEDLINE | ||
| 520 | |a © The Author(s) 2023. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissionsoup.com. | ||
| 520 | |a OBJECTIVE: Applications of machine learning in healthcare are of high interest and have the potential to improve patient care. Yet, the real-world accuracy of these models in clinical practice and on different patient subpopulations remains unclear. To address these important questions, we hosted a community challenge to evaluate methods that predict healthcare outcomes. We focused on the prediction of all-cause mortality as the community challenge question | ||
| 520 | |a MATERIALS AND METHODS: Using a Model-to-Data framework, 345 registered participants, coalescing into 25 independent teams, spread over 3 continents and 10 countries, generated 25 accurate models all trained on a dataset of over 1.1 million patients and evaluated on patients prospectively collected over a 1-year observation of a large health system | ||
| 520 | |a RESULTS: The top performing team achieved a final area under the receiver operator curve of 0.947 (95% CI, 0.942-0.951) and an area under the precision-recall curve of 0.487 (95% CI, 0.458-0.499) on a prospectively collected patient cohort | ||
| 520 | |a DISCUSSION: Post hoc analysis after the challenge revealed that models differ in accuracy on subpopulations, delineated by race or gender, even when they are trained on the same data | ||
| 520 | |a CONCLUSION: This is the largest community challenge focused on the evaluation of state-of-the-art machine learning methods in a healthcare system performed to date, revealing both opportunities and pitfalls of clinical AI | ||
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a Research Support, N.I.H., Extramural | |
| 650 | 4 | |a evaluation | |
| 650 | 4 | |a health informatics | |
| 650 | 4 | |a machine learning | |
| 700 | 1 | |a Schaffter, Thomas |e verfasserin |4 aut | |
| 700 | 1 | |a Yan, Yao |e verfasserin |4 aut | |
| 700 | 1 | |a Yu, Thomas |e verfasserin |4 aut | |
| 700 | 1 | |a Prosser, Justin |e verfasserin |4 aut | |
| 700 | 1 | |a Gao, Jifan |e verfasserin |4 aut | |
| 700 | 1 | |a Chen, Guanhua |e verfasserin |4 aut | |
| 700 | 1 | |a Charzewski, Łukasz |e verfasserin |4 aut | |
| 700 | 1 | |a Nawalany, Zofia |e verfasserin |4 aut | |
| 700 | 1 | |a Brugere, Ivan |e verfasserin |4 aut | |
| 700 | 1 | |a Retkute, Renata |e verfasserin |4 aut | |
| 700 | 1 | |a Prusokas, Alidivinas |e verfasserin |4 aut | |
| 700 | 1 | |a Prusokas, Augustinas |e verfasserin |4 aut | |
| 700 | 1 | |a Choi, Yonghwa |e verfasserin |4 aut | |
| 700 | 1 | |a Lee, Sanghoon |e verfasserin |4 aut | |
| 700 | 1 | |a Choe, Junseok |e verfasserin |4 aut | |
| 700 | 1 | |a Lee, Inggeol |e verfasserin |4 aut | |
| 700 | 1 | |a Kim, Sunkyu |e verfasserin |4 aut | |
| 700 | 1 | |a Kang, Jaewoo |e verfasserin |4 aut | |
| 700 | 1 | |a Mooney, Sean D |e verfasserin |4 aut | |
| 700 | 1 | |a Guinney, Justin |e verfasserin |4 aut | |
| 700 | 0 | |a Patient Mortality Prediction DREAM Challenge Consortium |e verfasserin |4 aut | |
| 700 | 1 | |a Lee, Aaron |e investigator |4 oth | |
| 700 | 1 | |a Salehzadeh-Yazdi, Ali |e investigator |4 oth | |
| 700 | 1 | |a Prusokas, Alidivinas |e investigator |4 oth | |
| 700 | 1 | |a Basu, Anand |e investigator |4 oth | |
| 700 | 1 | |a Belouali, Anas |e investigator |4 oth | |
| 700 | 1 | |a Becker, Ann-Kristin |e investigator |4 oth | |
| 700 | 1 | |a Israel, Ariel |e investigator |4 oth | |
| 700 | 1 | |a Prusokas, Augustinas |e investigator |4 oth | |
| 700 | 1 | |a Winter, B |e investigator |4 oth | |
| 700 | 1 | |a Moreno, Carlos Vega |e investigator |4 oth | |
| 700 | 1 | |a Kurz, Christoph |e investigator |4 oth | |
| 700 | 1 | |a Waltemath, Dagmar |e investigator |4 oth | |
| 700 | 1 | |a Schweinoch, Darius |e investigator |4 oth | |
| 700 | 1 | |a Glaab, Enrico |e investigator |4 oth | |
| 700 | 1 | |a Luo, Gang |e investigator |4 oth | |
| 700 | 1 | |a Chen, Guanhua |e investigator |4 oth | |
| 700 | 1 | |a Zacharias, Helena U |e investigator |4 oth | |
| 700 | 1 | |a Qiao, Hezhe |e investigator |4 oth | |
| 700 | 1 | |a Lee, Inggeol |e investigator |4 oth | |
| 700 | 1 | |a Brugere, Ivan |e investigator |4 oth | |
| 700 | 1 | |a Kang, Jaewoo |e investigator |4 oth | |
| 700 | 1 | |a Gao, Jifan |e investigator |4 oth | |
| 700 | 1 | |a Truthmann, Julia |e investigator |4 oth | |
| 700 | 1 | |a Choe, JunSeok |e investigator |4 oth | |
| 700 | 1 | |a Stephens, Kari A |e investigator |4 oth | |
| 700 | 1 | |a Kaderali, Lars |e investigator |4 oth | |
| 700 | 1 | |a Varshney, Lav R |e investigator |4 oth | |
| 700 | 1 | |a Vollmer, Marcus |e investigator |4 oth | |
| 700 | 1 | |a Pandi, Maria-Theodora |e investigator |4 oth | |
| 700 | 1 | |a Gunn, Martin L |e investigator |4 oth | |
| 700 | 1 | |a Yetisgen, Meliha |e investigator |4 oth | |
| 700 | 1 | |a Nath, Neetika |e investigator |4 oth | |
| 700 | 1 | |a Hammarlund, Noah |e investigator |4 oth | |
| 700 | 1 | |a Müller-Stricker, Oliver |e investigator |4 oth | |
| 700 | 1 | |a Togias, Panagiotis |e investigator |4 oth | |
| 700 | 1 | |a Heagerty, Patrick J |e investigator |4 oth | |
| 700 | 1 | |a Muir, Peter |e investigator |4 oth | |
| 700 | 1 | |a Banda, Peter |e investigator |4 oth | |
| 700 | 1 | |a Retkute, Renata |e investigator |4 oth | |
| 700 | 1 | |a Henkel, Ron |e investigator |4 oth | |
| 700 | 1 | |a Madgi, Sagar |e investigator |4 oth | |
| 700 | 1 | |a Gupta, Samir |e investigator |4 oth | |
| 700 | 1 | |a Lee, Sanghoon |e investigator |4 oth | |
| 700 | 1 | |a Mooney, Sean |e investigator |4 oth | |
| 700 | 1 | |a Kannattikuni, Shabeeb |e investigator |4 oth | |
| 700 | 1 | |a Sarhadi, Shamim |e investigator |4 oth | |
| 700 | 1 | |a Omar, Shikhar |e investigator |4 oth | |
| 700 | 1 | |a Wang, Shuo |e investigator |4 oth | |
| 700 | 1 | |a Ghosh, Soumyabrata |e investigator |4 oth | |
| 700 | 1 | |a Neumann, Stefan |e investigator |4 oth | |
| 700 | 1 | |a Simm, Stefan |e investigator |4 oth | |
| 700 | 1 | |a Madhavan, Subha |e investigator |4 oth | |
| 700 | 1 | |a Kim, Sunkyu |e investigator |4 oth | |
| 700 | 1 | |a Von Yu, Thomas |e investigator |4 oth | |
| 700 | 1 | |a Satagopam, Venkata |e investigator |4 oth | |
| 700 | 1 | |a Pejaver, Vikas |e investigator |4 oth | |
| 700 | 1 | |a Gupta, Yachee |e investigator |4 oth | |
| 700 | 1 | |a Choi, Yonghwa |e investigator |4 oth | |
| 700 | 1 | |a Nawalany, Zofia |e investigator |4 oth | |
| 700 | 1 | |a Charzewski, Łukasz |e investigator |4 oth | |
| 700 | 1 | |a Lee, Aaron |e investigator |4 oth | |
| 700 | 1 | |a Salehzadeh-Yazdi, Ali |e investigator |4 oth | |
| 700 | 1 | |a Prusokas, Alidivinas |e investigator |4 oth | |
| 700 | 1 | |a Basu, Anand |e investigator |4 oth | |
| 700 | 1 | |a Belouali, Anas |e investigator |4 oth | |
| 700 | 1 | |a Becker, Ann-Kristin |e investigator |4 oth | |
| 700 | 1 | |a Israel, Ariel |e investigator |4 oth | |
| 700 | 1 | |a Prusokas, Augustinas |e investigator |4 oth | |
| 700 | 1 | |a Winter, B |e investigator |4 oth | |
| 700 | 1 | |a Moreno, Carlos Vega |e investigator |4 oth | |
| 700 | 1 | |a Kurz, Christoph |e investigator |4 oth | |
| 700 | 1 | |a Waltemath, Dagmar |e investigator |4 oth | |
| 700 | 1 | |a Schweinoch, Darius |e investigator |4 oth | |
| 700 | 1 | |a Glaab, Enrico |e investigator |4 oth | |
| 700 | 1 | |a Luo, Gang |e investigator |4 oth | |
| 700 | 1 | |a Chen, Guanhua |e investigator |4 oth | |
| 700 | 1 | |a Zacharias, Helena U |e investigator |4 oth | |
| 700 | 1 | |a Qiao, Hezhe |e investigator |4 oth | |
| 700 | 1 | |a Lee, Inggeol |e investigator |4 oth | |
| 700 | 1 | |a Brugere, Ivan |e investigator |4 oth | |
| 700 | 1 | |a Kang, Jaewoo |e investigator |4 oth | |
| 700 | 1 | |a Gao, Jifan |e investigator |4 oth | |
| 700 | 1 | |a Truthmann, Julia |e investigator |4 oth | |
| 700 | 1 | |a Choe, JunSeok |e investigator |4 oth | |
| 700 | 1 | |a Stephens, Kari A |e investigator |4 oth | |
| 700 | 1 | |a Kaderali, Lars |e investigator |4 oth | |
| 700 | 1 | |a Varshney, Lav R |e investigator |4 oth | |
| 700 | 1 | |a Vollmer, Marcus |e investigator |4 oth | |
| 700 | 1 | |a Pandi, Maria-Theodora |e investigator |4 oth | |
| 700 | 1 | |a Gunn, Martin L |e investigator |4 oth | |
| 700 | 1 | |a Yetisgen, Meliha |e investigator |4 oth | |
| 700 | 1 | |a Nath, Neetika |e investigator |4 oth | |
| 700 | 1 | |a Hammarlund, Noah |e investigator |4 oth | |
| 700 | 1 | |a Müller-Stricker, Oliver |e investigator |4 oth | |
| 700 | 1 | |a Togias, Panagiotis |e investigator |4 oth | |
| 700 | 1 | |a Heagerty, Patrick J |e investigator |4 oth | |
| 700 | 1 | |a Muir, Peter |e investigator |4 oth | |
| 700 | 1 | |a Banda, Peter |e investigator |4 oth | |
| 700 | 1 | |a Retkute, Renata |e investigator |4 oth | |
| 700 | 1 | |a Henkel, Ron |e investigator |4 oth | |
| 773 | 0 | 8 | |i Enthalten in |t Journal of the American Medical Informatics Association : JAMIA |d 1997 |g 31(2023), 1 vom: 22. Dez., Seite 35-44 |w (DE-627)NLM074735535 |x 1527-974X |7 nnns |
| 773 | 1 | 8 | |g volume:31 |g year:2023 |g number:1 |g day:22 |g month:12 |g pages:35-44 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1093/jamia/ocad159 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 31 |j 2023 |e 1 |b 22 |c 12 |h 35-44 | ||