Statistically significant differences versus convincing evidence of real treatment effects : an analysis of the false positive risk for single-centre trials in anaesthesia
Copyright © 2023 The Author(s). Published by Elsevier Ltd.. All rights reserved..
BACKGROUND: The American Statistical Association has highlighted problems with null hypothesis significance testing and outlined alternative approaches that may 'supplement or even replace P-values'. One alternative is to report the false positive risk (FPR), which quantifies the chance the null hypothesis is true when the result is statistically significant.
METHODS: We reviewed single-centre, randomised trials in 10 anaesthesia journals over 6 yr where differences in a primary binary outcome were statistically significant. We calculated a Bayes factor by two methods (Gunel, Kass). From the Bayes factor we calculated the FPR for different prior beliefs for a real treatment effect. Prior beliefs were quantified by assigning pretest probabilities to the null and alternative hypotheses.
RESULTS: For equal pretest probabilities of 0.5, the median (inter-quartile range [IQR]) FPR was 6% (1-22%) by the Gunel method and 6% (1-19%) by the Kass method. One in five trials had an FPR ≥20%. For trials reporting P-values 0.01-0.05, the median (IQR) FPR was 25% (16-30%) by the Gunel method and 20% (16-25%) by the Kass method. More than 90% of trials reporting P-values 0.01-0.05 required a pretest probability >0.5 to achieve an FPR of 5%. The median (IQR) difference in the FPR calculated by the two methods was 0% (0-2%).
CONCLUSIONS: Our findings suggest that a substantial proportion of single-centre trials in anaesthesia reporting statistically significant differences provide limited evidence of real treatment effects, or, alternatively, required an implausibly high prior belief in a real treatment effect.
CLINICAL TRIAL REGISTRATION: PROSPERO (CRD42023350783).
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2024 |
|---|---|
| Erschienen: |
2024 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:132 |
|---|---|
| Enthalten in: |
British journal of anaesthesia - 132(2024), 1 vom: 29. Jan., Seite 116-123 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Sidebotham, David [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
Anaesthesia |
|---|
| Anmerkungen: |
Date Completed 05.01.2024 Date Revised 05.01.2024 published: Print-Electronic Citation Status MEDLINE |
|---|
| doi: |
10.1016/j.bja.2023.10.036 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM365216933 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM365216933 | ||
| 003 | DE-627 | ||
| 005 | 20240108142147.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231226s2024 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1016/j.bja.2023.10.036 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1249.xml |
| 035 | |a (DE-627)NLM365216933 | ||
| 035 | |a (NLM)38030552 | ||
| 035 | |a (PII)S0007-0912(23)00616-5 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Sidebotham, David |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Statistically significant differences versus convincing evidence of real treatment effects |b an analysis of the false positive risk for single-centre trials in anaesthesia |
| 264 | 1 | |c 2024 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 05.01.2024 | ||
| 500 | |a Date Revised 05.01.2024 | ||
| 500 | |a published: Print-Electronic | ||
| 500 | |a Citation Status MEDLINE | ||
| 520 | |a Copyright © 2023 The Author(s). Published by Elsevier Ltd.. All rights reserved. | ||
| 520 | |a BACKGROUND: The American Statistical Association has highlighted problems with null hypothesis significance testing and outlined alternative approaches that may 'supplement or even replace P-values'. One alternative is to report the false positive risk (FPR), which quantifies the chance the null hypothesis is true when the result is statistically significant | ||
| 520 | |a METHODS: We reviewed single-centre, randomised trials in 10 anaesthesia journals over 6 yr where differences in a primary binary outcome were statistically significant. We calculated a Bayes factor by two methods (Gunel, Kass). From the Bayes factor we calculated the FPR for different prior beliefs for a real treatment effect. Prior beliefs were quantified by assigning pretest probabilities to the null and alternative hypotheses | ||
| 520 | |a RESULTS: For equal pretest probabilities of 0.5, the median (inter-quartile range [IQR]) FPR was 6% (1-22%) by the Gunel method and 6% (1-19%) by the Kass method. One in five trials had an FPR ≥20%. For trials reporting P-values 0.01-0.05, the median (IQR) FPR was 25% (16-30%) by the Gunel method and 20% (16-25%) by the Kass method. More than 90% of trials reporting P-values 0.01-0.05 required a pretest probability >0.5 to achieve an FPR of 5%. The median (IQR) difference in the FPR calculated by the two methods was 0% (0-2%) | ||
| 520 | |a CONCLUSIONS: Our findings suggest that a substantial proportion of single-centre trials in anaesthesia reporting statistically significant differences provide limited evidence of real treatment effects, or, alternatively, required an implausibly high prior belief in a real treatment effect | ||
| 520 | |a CLINICAL TRIAL REGISTRATION: PROSPERO (CRD42023350783) | ||
| 650 | 4 | |a Review | |
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a Bayes factor | |
| 650 | 4 | |a Bayes' theorem | |
| 650 | 4 | |a anaesthesia | |
| 650 | 4 | |a false positive risk | |
| 650 | 4 | |a research design | |
| 650 | 4 | |a sample size | |
| 650 | 4 | |a significance testing | |
| 700 | 1 | |a Dominick, Felicity |e verfasserin |4 aut | |
| 700 | 1 | |a Deng, Carolyn |e verfasserin |4 aut | |
| 700 | 1 | |a Barlow, Jake |e verfasserin |4 aut | |
| 700 | 1 | |a Jones, Philip M |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t British journal of anaesthesia |d 1946 |g 132(2024), 1 vom: 29. Jan., Seite 116-123 |w (DE-627)NLM000000310 |x 1471-6771 |7 nnns |
| 773 | 1 | 8 | |g volume:132 |g year:2024 |g number:1 |g day:29 |g month:01 |g pages:116-123 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1016/j.bja.2023.10.036 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 132 |j 2024 |e 1 |b 29 |c 01 |h 116-123 | ||