A systematic review of conference papers presented at two large Japanese psychology conferences in 2013 and 2018 : did Japanese social psychologists selectively report p < 0.05 results without peer review?
© 2024 Hiraishi et al..
We conducted a systematic review of conference papers in social psychology at two large psychology conferences in Japan: the Japanese Psychological Association and the Japanese Society for Social Psychology. The conference papers were effectively not subjected to peer review; hence, they were suitable for testing if psychologists selectively reported statistically significant findings without pressure from journal editors and reviewers. We investigated the distributions of z-values converted from the p-values reported in the articles presented at the 2013 and 2018 conferences. The z-curve analyses suggest the existence of selective reporting by the authors in 2013. The expected discovery rate (EDR) was much lower than the observed discovery rate (ODR; 7% vs. 76%, respectively), and the 95% confidence interval (CI) did not include the ODR. However, this does not mean that the set of studies completely lacked evidential value. The expected replication rate (ERR) was 31%; this is significantly higher than 5%, which was expected under the null hypothesis of no effect. Changes were observed between 2013 and 2018. The ERR increased (31% to 44%), and the EDR almost doubled (7% to 13%). However, the estimation of the maximum false discovery rate (FDR; 68% in 2013 and 35% in 2018) suggested that a substantial proportion of the reported findings were false positives. Overall, while social psychologists in Japan engaged in selective reporting, this does not mean that the entire field was covered with false positives. In addition, slight signs of improvement were observed in how they reported their findings. Still, the evidential value of the target studies was weak, even in 2018, allowing for no optimism.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2024 |
|---|---|
| Erschienen: |
2024 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:12 |
|---|---|
| Enthalten in: |
PeerJ - 12(2024) vom: 11., Seite e16763 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Hiraishi, Kai [VerfasserIn] |
|---|
| Links: |
|---|
| Anmerkungen: |
Date Completed 23.01.2024 Date Revised 23.01.2024 published: Electronic-eCollection Citation Status MEDLINE |
|---|
| doi: |
10.7717/peerj.16763 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM367412225 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM367412225 | ||
| 003 | DE-627 | ||
| 005 | 20240123232102.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 240122s2024 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.7717/peerj.16763 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1268.xml |
| 035 | |a (DE-627)NLM367412225 | ||
| 035 | |a (NLM)38250729 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Hiraishi, Kai |e verfasserin |4 aut | |
| 245 | 1 | 2 | |a A systematic review of conference papers presented at two large Japanese psychology conferences in 2013 and 2018 |b did Japanese social psychologists selectively report p < 0.05 results without peer review? |
| 264 | 1 | |c 2024 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 23.01.2024 | ||
| 500 | |a Date Revised 23.01.2024 | ||
| 500 | |a published: Electronic-eCollection | ||
| 500 | |a Citation Status MEDLINE | ||
| 520 | |a © 2024 Hiraishi et al. | ||
| 520 | |a We conducted a systematic review of conference papers in social psychology at two large psychology conferences in Japan: the Japanese Psychological Association and the Japanese Society for Social Psychology. The conference papers were effectively not subjected to peer review; hence, they were suitable for testing if psychologists selectively reported statistically significant findings without pressure from journal editors and reviewers. We investigated the distributions of z-values converted from the p-values reported in the articles presented at the 2013 and 2018 conferences. The z-curve analyses suggest the existence of selective reporting by the authors in 2013. The expected discovery rate (EDR) was much lower than the observed discovery rate (ODR; 7% vs. 76%, respectively), and the 95% confidence interval (CI) did not include the ODR. However, this does not mean that the set of studies completely lacked evidential value. The expected replication rate (ERR) was 31%; this is significantly higher than 5%, which was expected under the null hypothesis of no effect. Changes were observed between 2013 and 2018. The ERR increased (31% to 44%), and the EDR almost doubled (7% to 13%). However, the estimation of the maximum false discovery rate (FDR; 68% in 2013 and 35% in 2018) suggested that a substantial proportion of the reported findings were false positives. Overall, while social psychologists in Japan engaged in selective reporting, this does not mean that the entire field was covered with false positives. In addition, slight signs of improvement were observed in how they reported their findings. Still, the evidential value of the target studies was weak, even in 2018, allowing for no optimism | ||
| 650 | 4 | |a Systematic Review | |
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a Evidential value | |
| 650 | 4 | |a Meta-analysis | |
| 650 | 4 | |a Publicaiton bias | |
| 650 | 4 | |a Questionable research practices | |
| 650 | 4 | |a Replicability crisis in psychology | |
| 650 | 4 | |a Selective reporting | |
| 650 | 4 | |a Systematic review | |
| 650 | 4 | |a p-curve analysis | |
| 650 | 4 | |a z-curve analysis | |
| 700 | 1 | |a Miura, Asako |e verfasserin |4 aut | |
| 700 | 1 | |a Higuchi, Masataka |e verfasserin |4 aut | |
| 700 | 1 | |a Fujishima, Yoshitsugu |e verfasserin |4 aut | |
| 700 | 1 | |a Nakamura, Daiki |e verfasserin |4 aut | |
| 700 | 1 | |a Suyama, Masaki |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t PeerJ |d 2013 |g 12(2024) vom: 11., Seite e16763 |w (DE-627)NLM227171578 |x 2167-8359 |7 nnns |
| 773 | 1 | 8 | |g volume:12 |g year:2024 |g day:11 |g pages:e16763 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.7717/peerj.16763 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 12 |j 2024 |b 11 |h e16763 | ||