Confidence score : a data-driven measure for inclusive systematic reviews considering unpublished preprints
© The Author(s) 2023. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissionsoup.com..
OBJECTIVES: COVID-19, since its emergence in December 2019, has globally impacted research. Over 360 000 COVID-19-related manuscripts have been published on PubMed and preprint servers like medRxiv and bioRxiv, with preprints comprising about 15% of all manuscripts. Yet, the role and impact of preprints on COVID-19 research and evidence synthesis remain uncertain.
MATERIALS AND METHODS: We propose a novel data-driven method for assigning weights to individual preprints in systematic reviews and meta-analyses. This weight termed the "confidence score" is obtained using the survival cure model, also known as the survival mixture model, which takes into account the time elapsed between posting and publication of a preprint, as well as metadata such as the number of first 2-week citations, sample size, and study type.
RESULTS: Using 146 preprints on COVID-19 therapeutics posted from the beginning of the pandemic through April 30, 2021, we validated the confidence scores, showing an area under the curve of 0.95 (95% CI, 0.92-0.98). Through a use case on the effectiveness of hydroxychloroquine, we demonstrated how these scores can be incorporated practically into meta-analyses to properly weigh preprints.
DISCUSSION: It is important to note that our method does not aim to replace existing measures of study quality but rather serves as a supplementary measure that overcomes some limitations of current approaches.
CONCLUSION: Our proposed confidence score has the potential to improve systematic reviews of evidence related to COVID-19 and other clinical conditions by providing a data-driven approach to including unpublished manuscripts.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2024 |
|---|---|
| Erschienen: |
2024 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:31 |
|---|---|
| Enthalten in: |
Journal of the American Medical Informatics Association : JAMIA - 31(2024), 4 vom: 03. Apr., Seite 809-819 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Tong, Jiayi [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
Data-driven modeling |
|---|
| Anmerkungen: |
Date Completed 05.04.2024 Date Revised 05.04.2024 published: Print Citation Status MEDLINE |
|---|
| doi: |
10.1093/jamia/ocad248 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM365565105 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM365565105 | ||
| 003 | DE-627 | ||
| 005 | 20240405233256.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231226s2024 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1093/jamia/ocad248 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1366.xml |
| 035 | |a (DE-627)NLM365565105 | ||
| 035 | |a (NLM)38065694 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Tong, Jiayi |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Confidence score |b a data-driven measure for inclusive systematic reviews considering unpublished preprints |
| 264 | 1 | |c 2024 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 05.04.2024 | ||
| 500 | |a Date Revised 05.04.2024 | ||
| 500 | |a published: Print | ||
| 500 | |a Citation Status MEDLINE | ||
| 520 | |a © The Author(s) 2023. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissionsoup.com. | ||
| 520 | |a OBJECTIVES: COVID-19, since its emergence in December 2019, has globally impacted research. Over 360 000 COVID-19-related manuscripts have been published on PubMed and preprint servers like medRxiv and bioRxiv, with preprints comprising about 15% of all manuscripts. Yet, the role and impact of preprints on COVID-19 research and evidence synthesis remain uncertain | ||
| 520 | |a MATERIALS AND METHODS: We propose a novel data-driven method for assigning weights to individual preprints in systematic reviews and meta-analyses. This weight termed the "confidence score" is obtained using the survival cure model, also known as the survival mixture model, which takes into account the time elapsed between posting and publication of a preprint, as well as metadata such as the number of first 2-week citations, sample size, and study type | ||
| 520 | |a RESULTS: Using 146 preprints on COVID-19 therapeutics posted from the beginning of the pandemic through April 30, 2021, we validated the confidence scores, showing an area under the curve of 0.95 (95% CI, 0.92-0.98). Through a use case on the effectiveness of hydroxychloroquine, we demonstrated how these scores can be incorporated practically into meta-analyses to properly weigh preprints | ||
| 520 | |a DISCUSSION: It is important to note that our method does not aim to replace existing measures of study quality but rather serves as a supplementary measure that overcomes some limitations of current approaches | ||
| 520 | |a CONCLUSION: Our proposed confidence score has the potential to improve systematic reviews of evidence related to COVID-19 and other clinical conditions by providing a data-driven approach to including unpublished manuscripts | ||
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a data-driven modeling | |
| 650 | 4 | |a evidence synthesis | |
| 650 | 4 | |a preprint | |
| 650 | 4 | |a systematic review | |
| 700 | 1 | |a Luo, Chongliang |e verfasserin |4 aut | |
| 700 | 1 | |a Sun, Yifei |e verfasserin |4 aut | |
| 700 | 1 | |a Duan, Rui |e verfasserin |4 aut | |
| 700 | 1 | |a Saine, M Elle |e verfasserin |4 aut | |
| 700 | 1 | |a Lin, Lifeng |e verfasserin |4 aut | |
| 700 | 1 | |a Peng, Yifan |e verfasserin |4 aut | |
| 700 | 1 | |a Lu, Yiwen |e verfasserin |4 aut | |
| 700 | 1 | |a Batra, Anchita |e verfasserin |4 aut | |
| 700 | 1 | |a Pan, Anni |e verfasserin |4 aut | |
| 700 | 1 | |a Wang, Olivia |e verfasserin |4 aut | |
| 700 | 1 | |a Li, Ruowang |e verfasserin |4 aut | |
| 700 | 1 | |a Marks-Anglin, Arielle |e verfasserin |4 aut | |
| 700 | 1 | |a Yang, Yuchen |e verfasserin |4 aut | |
| 700 | 1 | |a Zuo, Xu |e verfasserin |4 aut | |
| 700 | 1 | |a Liu, Yulun |e verfasserin |4 aut | |
| 700 | 1 | |a Bian, Jiang |e verfasserin |4 aut | |
| 700 | 1 | |a Kimmel, Stephen E |e verfasserin |4 aut | |
| 700 | 1 | |a Hamilton, Keith |e verfasserin |4 aut | |
| 700 | 1 | |a Cuker, Adam |e verfasserin |4 aut | |
| 700 | 1 | |a Hubbard, Rebecca A |e verfasserin |4 aut | |
| 700 | 1 | |a Xu, Hua |e verfasserin |4 aut | |
| 700 | 1 | |a Chen, Yong |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t Journal of the American Medical Informatics Association : JAMIA |d 1997 |g 31(2024), 4 vom: 03. Apr., Seite 809-819 |w (DE-627)NLM074735535 |x 1527-974X |7 nnns |
| 773 | 1 | 8 | |g volume:31 |g year:2024 |g number:4 |g day:03 |g month:04 |g pages:809-819 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1093/jamia/ocad248 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 31 |j 2024 |e 4 |b 03 |c 04 |h 809-819 | ||