Meta-research : How many diagnostic or prognostic models published in radiological journals are evaluated externally?
© 2023. The Author(s)..
OBJECTIVES: Prognostic and diagnostic models must work in their intended clinical setting, proven via "external evaluation", preferably by authors uninvolved with model development. By systematic review, we determined the proportion of models published in high-impact radiological journals that are evaluated subsequently.
METHODS: We hand-searched three radiological journals for multivariable diagnostic/prognostic models 2013-2015 inclusive, developed using regression. We assessed completeness of data presentation to allow subsequent external evaluation. We then searched literature to August 2022 to identify external evaluations of these index models.
RESULTS: We identified 98 index studies (73 prognostic; 25 diagnostic) describing 145 models. Only 15 (15%) index studies presented an evaluation (two external). No model was updated. Only 20 (20%) studies presented a model equation. Just 7 (15%) studies developing Cox models presented a risk table, and just 4 (9%) presented the baseline hazard. Two (4%) studies developing non-Cox models presented the intercept. Just 20 (20%) articles presented a Kaplan-Meier curve of the final model. The 98 index studies attracted 4224 citations (including 559 self-citations), median 28 per study. We identified just six (6%) subsequent external evaluations of an index model, five of which were external evaluations by researchers uninvolved with model development, and from a different institution.
CONCLUSIONS: Very few prognostic or diagnostic models published in radiological literature are evaluated externally, suggesting wasted research effort and resources. Authors' published models should present data sufficient to allow external evaluation by others. To achieve clinical utility, researchers should concentrate on model evaluation and updating rather than continual redevelopment.
CLINICAL RELEVANCE STATEMENT: The large majority of prognostic and diagnostic models published in high-impact radiological journals are never evaluated. It would be more efficient for researchers to evaluate existing models rather than practice continual redevelopment.
KEY POINTS: • Systematic review of highly cited radiological literature identified few diagnostic or prognostic models that were evaluated subsequently by researchers uninvolved with the original model. • Published radiological models frequently omit important information necessary for others to perform an external evaluation: Only 20% of studies presented a model equation or nomogram. • A large proportion of research citing published models focuses on redevelopment and ignores evaluation and updating, which would be a more efficient use of research resources.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2024 |
|---|---|
| Erschienen: |
2024 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:34 |
|---|---|
| Enthalten in: |
European radiology - 34(2024), 4 vom: 12. Apr., Seite 2524-2533 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Hameed, Maira [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
Evaluation study |
|---|
| Anmerkungen: |
Date Completed 22.03.2024 Date Revised 09.04.2024 published: Print-Electronic Citation Status MEDLINE |
|---|
| doi: |
10.1007/s00330-023-10168-3 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM361943733 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM361943733 | ||
| 003 | DE-627 | ||
| 005 | 20240409232150.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231226s2024 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1007/s00330-023-10168-3 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1370.xml |
| 035 | |a (DE-627)NLM361943733 | ||
| 035 | |a (NLM)37696974 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Hameed, Maira |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Meta-research |b How many diagnostic or prognostic models published in radiological journals are evaluated externally? |
| 264 | 1 | |c 2024 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 22.03.2024 | ||
| 500 | |a Date Revised 09.04.2024 | ||
| 500 | |a published: Print-Electronic | ||
| 500 | |a Citation Status MEDLINE | ||
| 520 | |a © 2023. The Author(s). | ||
| 520 | |a OBJECTIVES: Prognostic and diagnostic models must work in their intended clinical setting, proven via "external evaluation", preferably by authors uninvolved with model development. By systematic review, we determined the proportion of models published in high-impact radiological journals that are evaluated subsequently | ||
| 520 | |a METHODS: We hand-searched three radiological journals for multivariable diagnostic/prognostic models 2013-2015 inclusive, developed using regression. We assessed completeness of data presentation to allow subsequent external evaluation. We then searched literature to August 2022 to identify external evaluations of these index models | ||
| 520 | |a RESULTS: We identified 98 index studies (73 prognostic; 25 diagnostic) describing 145 models. Only 15 (15%) index studies presented an evaluation (two external). No model was updated. Only 20 (20%) studies presented a model equation. Just 7 (15%) studies developing Cox models presented a risk table, and just 4 (9%) presented the baseline hazard. Two (4%) studies developing non-Cox models presented the intercept. Just 20 (20%) articles presented a Kaplan-Meier curve of the final model. The 98 index studies attracted 4224 citations (including 559 self-citations), median 28 per study. We identified just six (6%) subsequent external evaluations of an index model, five of which were external evaluations by researchers uninvolved with model development, and from a different institution | ||
| 520 | |a CONCLUSIONS: Very few prognostic or diagnostic models published in radiological literature are evaluated externally, suggesting wasted research effort and resources. Authors' published models should present data sufficient to allow external evaluation by others. To achieve clinical utility, researchers should concentrate on model evaluation and updating rather than continual redevelopment | ||
| 520 | |a CLINICAL RELEVANCE STATEMENT: The large majority of prognostic and diagnostic models published in high-impact radiological journals are never evaluated. It would be more efficient for researchers to evaluate existing models rather than practice continual redevelopment | ||
| 520 | |a KEY POINTS: • Systematic review of highly cited radiological literature identified few diagnostic or prognostic models that were evaluated subsequently by researchers uninvolved with the original model. • Published radiological models frequently omit important information necessary for others to perform an external evaluation: Only 20% of studies presented a model equation or nomogram. • A large proportion of research citing published models focuses on redevelopment and ignores evaluation and updating, which would be a more efficient use of research resources | ||
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a Evaluation study | |
| 650 | 4 | |a Logistic models | |
| 650 | 4 | |a Models statistical | |
| 650 | 4 | |a Prognosis | |
| 650 | 4 | |a Proportional hazards models | |
| 700 | 1 | |a Yeung, Jason |e verfasserin |4 aut | |
| 700 | 1 | |a Boone, Darren |e verfasserin |4 aut | |
| 700 | 1 | |a Mallett, Sue |e verfasserin |4 aut | |
| 700 | 1 | |a Halligan, Steve |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t European radiology |d 1991 |g 34(2024), 4 vom: 12. Apr., Seite 2524-2533 |w (DE-627)NLM087691310 |x 1432-1084 |7 nnns |
| 773 | 1 | 8 | |g volume:34 |g year:2024 |g number:4 |g day:12 |g month:04 |g pages:2524-2533 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1007/s00330-023-10168-3 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 34 |j 2024 |e 4 |b 12 |c 04 |h 2524-2533 | ||