Automated medical literature screening using artificial intelligence : a systematic review and meta-analysis
© The Author(s) 2022. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissionsoup.com..
OBJECTIVE: We aim to investigate the application and accuracy of artificial intelligence (AI) methods for automated medical literature screening for systematic reviews.
MATERIALS AND METHODS: We systematically searched PubMed, Embase, and IEEE Xplore Digital Library to identify potentially relevant studies. We included studies in automated literature screening that reported study question, source of dataset, and developed algorithm models for literature screening. The literature screening results by human investigators were considered to be the reference standard. Quantitative synthesis of the accuracy was conducted using a bivariate model.
RESULTS: Eighty-six studies were included in our systematic review and 17 studies were further included for meta-analysis. The combined recall, specificity, and precision were 0.928 [95% confidence interval (CI), 0.878-0.958], 0.647 (95% CI, 0.442-0.809), and 0.200 (95% CI, 0.135-0.287) when achieving maximized recall, but were 0.708 (95% CI, 0.570-0.816), 0.921 (95% CI, 0.824-0.967), and 0.461 (95% CI, 0.375-0.549) when achieving maximized precision in the AI models. No significant difference was found in recall among subgroup analyses including the algorithms, the number of screened literatures, and the fraction of included literatures.
DISCUSSION AND CONCLUSION: This systematic review and meta-analysis study showed that the recall is more important than the specificity or precision in literature screening, and a recall over 0.95 should be prioritized. We recommend to report the effectiveness indices of automatic algorithms separately. At the current stage manual literature screening is still indispensable for medical systematic reviews.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2022 |
|---|---|
| Erschienen: |
2022 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:29 |
|---|---|
| Enthalten in: |
Journal of the American Medical Informatics Association : JAMIA - 29(2022), 8 vom: 12. Juli, Seite 1425-1432 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Feng, Yunying [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
Artificial intelligence |
|---|
| Anmerkungen: |
Date Completed 14.07.2022 Date Revised 01.06.2023 published: Print Citation Status MEDLINE |
|---|
| doi: |
10.1093/jamia/ocac066 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM341633038 |
|---|
| LEADER | 01000naa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM341633038 | ||
| 003 | DE-627 | ||
| 005 | 20231226012229.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231226s2022 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1093/jamia/ocac066 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1138.xml |
| 035 | |a (DE-627)NLM341633038 | ||
| 035 | |a (NLM)35641139 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Feng, Yunying |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Automated medical literature screening using artificial intelligence |b a systematic review and meta-analysis |
| 264 | 1 | |c 2022 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 14.07.2022 | ||
| 500 | |a Date Revised 01.06.2023 | ||
| 500 | |a published: Print | ||
| 500 | |a Citation Status MEDLINE | ||
| 520 | |a © The Author(s) 2022. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissionsoup.com. | ||
| 520 | |a OBJECTIVE: We aim to investigate the application and accuracy of artificial intelligence (AI) methods for automated medical literature screening for systematic reviews | ||
| 520 | |a MATERIALS AND METHODS: We systematically searched PubMed, Embase, and IEEE Xplore Digital Library to identify potentially relevant studies. We included studies in automated literature screening that reported study question, source of dataset, and developed algorithm models for literature screening. The literature screening results by human investigators were considered to be the reference standard. Quantitative synthesis of the accuracy was conducted using a bivariate model | ||
| 520 | |a RESULTS: Eighty-six studies were included in our systematic review and 17 studies were further included for meta-analysis. The combined recall, specificity, and precision were 0.928 [95% confidence interval (CI), 0.878-0.958], 0.647 (95% CI, 0.442-0.809), and 0.200 (95% CI, 0.135-0.287) when achieving maximized recall, but were 0.708 (95% CI, 0.570-0.816), 0.921 (95% CI, 0.824-0.967), and 0.461 (95% CI, 0.375-0.549) when achieving maximized precision in the AI models. No significant difference was found in recall among subgroup analyses including the algorithms, the number of screened literatures, and the fraction of included literatures | ||
| 520 | |a DISCUSSION AND CONCLUSION: This systematic review and meta-analysis study showed that the recall is more important than the specificity or precision in literature screening, and a recall over 0.95 should be prioritized. We recommend to report the effectiveness indices of automatic algorithms separately. At the current stage manual literature screening is still indispensable for medical systematic reviews | ||
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a Meta-Analysis | |
| 650 | 4 | |a Systematic Review | |
| 650 | 4 | |a Research Support, Non-U.S. Gov't | |
| 650 | 4 | |a artificial intelligence | |
| 650 | 4 | |a diagnostic test accuracy | |
| 650 | 4 | |a evidence-based medicine | |
| 650 | 4 | |a natural language process | |
| 650 | 4 | |a systematic review | |
| 700 | 1 | |a Liang, Siyu |e verfasserin |4 aut | |
| 700 | 1 | |a Zhang, Yuelun |e verfasserin |4 aut | |
| 700 | 1 | |a Chen, Shi |e verfasserin |4 aut | |
| 700 | 1 | |a Wang, Qing |e verfasserin |4 aut | |
| 700 | 1 | |a Huang, Tianze |e verfasserin |4 aut | |
| 700 | 1 | |a Sun, Feng |e verfasserin |4 aut | |
| 700 | 1 | |a Liu, Xiaoqing |e verfasserin |4 aut | |
| 700 | 1 | |a Zhu, Huijuan |e verfasserin |4 aut | |
| 700 | 1 | |a Pan, Hui |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t Journal of the American Medical Informatics Association : JAMIA |d 1997 |g 29(2022), 8 vom: 12. Juli, Seite 1425-1432 |w (DE-627)NLM074735535 |x 1527-974X |7 nnns |
| 773 | 1 | 8 | |g volume:29 |g year:2022 |g number:8 |g day:12 |g month:07 |g pages:1425-1432 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1093/jamia/ocac066 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 29 |j 2022 |e 8 |b 12 |c 07 |h 1425-1432 | ||