Predicting and reasoning about replicability using structured groups
© 2023 The Authors..
This paper explores judgements about the replicability of social and behavioural sciences research and what drives those judgements. Using a mixed methods approach, it draws on qualitative and quantitative data elicited from groups using a structured approach called the IDEA protocol ('investigate', 'discuss', 'estimate' and 'aggregate'). Five groups of five people with relevant domain expertise evaluated 25 research claims that were subject to at least one replication study. Participants assessed the probability that each of the 25 research claims would replicate (i.e. that a replication study would find a statistically significant result in the same direction as the original study) and described the reasoning behind those judgements. We quantitatively analysed possible correlates of predictive accuracy, including self-rated expertise and updating of judgements after feedback and discussion. We qualitatively analysed the reasoning data to explore the cues, heuristics and patterns of reasoning used by participants. Participants achieved 84% classification accuracy in predicting replicability. Those who engaged in a greater breadth of reasoning provided more accurate replicability judgements. Some reasons were more commonly invoked by more accurate participants, such as 'effect size' and 'reputation' (e.g. of the field of research). There was also some evidence of a relationship between statistical literacy and accuracy.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2023 |
|---|---|
| Erschienen: |
2023 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:10 |
|---|---|
| Enthalten in: |
Royal Society open science - 10(2023), 6 vom: 08. Juni, Seite 221553 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Wintle, Bonnie C [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
Expert judgement |
|---|
| Anmerkungen: |
Date Revised 06.11.2023 published: Electronic-eCollection figshare: 10.6084/m9.figshare.c.6662674 Citation Status PubMed-not-MEDLINE |
|---|
| doi: |
10.1098/rsos.221553 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM357951344 |
|---|
| LEADER | 01000naa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM357951344 | ||
| 003 | DE-627 | ||
| 005 | 20231226073713.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231226s2023 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1098/rsos.221553 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1193.xml |
| 035 | |a (DE-627)NLM357951344 | ||
| 035 | |a (NLM)37293358 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Wintle, Bonnie C |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Predicting and reasoning about replicability using structured groups |
| 264 | 1 | |c 2023 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Revised 06.11.2023 | ||
| 500 | |a published: Electronic-eCollection | ||
| 500 | |a figshare: 10.6084/m9.figshare.c.6662674 | ||
| 500 | |a Citation Status PubMed-not-MEDLINE | ||
| 520 | |a © 2023 The Authors. | ||
| 520 | |a This paper explores judgements about the replicability of social and behavioural sciences research and what drives those judgements. Using a mixed methods approach, it draws on qualitative and quantitative data elicited from groups using a structured approach called the IDEA protocol ('investigate', 'discuss', 'estimate' and 'aggregate'). Five groups of five people with relevant domain expertise evaluated 25 research claims that were subject to at least one replication study. Participants assessed the probability that each of the 25 research claims would replicate (i.e. that a replication study would find a statistically significant result in the same direction as the original study) and described the reasoning behind those judgements. We quantitatively analysed possible correlates of predictive accuracy, including self-rated expertise and updating of judgements after feedback and discussion. We qualitatively analysed the reasoning data to explore the cues, heuristics and patterns of reasoning used by participants. Participants achieved 84% classification accuracy in predicting replicability. Those who engaged in a greater breadth of reasoning provided more accurate replicability judgements. Some reasons were more commonly invoked by more accurate participants, such as 'effect size' and 'reputation' (e.g. of the field of research). There was also some evidence of a relationship between statistical literacy and accuracy | ||
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a expert judgement | |
| 650 | 4 | |a forecasting | |
| 650 | 4 | |a meta-research | |
| 650 | 4 | |a metascience | |
| 650 | 4 | |a mixed methods | |
| 650 | 4 | |a replication | |
| 700 | 1 | |a Smith, Eden T |e verfasserin |4 aut | |
| 700 | 1 | |a Bush, Martin |e verfasserin |4 aut | |
| 700 | 1 | |a Mody, Fallon |e verfasserin |4 aut | |
| 700 | 1 | |a Wilkinson, David P |e verfasserin |4 aut | |
| 700 | 1 | |a Hanea, Anca M |e verfasserin |4 aut | |
| 700 | 1 | |a Marcoci, Alexandru |e verfasserin |4 aut | |
| 700 | 1 | |a Fraser, Hannah |e verfasserin |4 aut | |
| 700 | 1 | |a Hemming, Victoria |e verfasserin |4 aut | |
| 700 | 1 | |a Thorn, Felix Singleton |e verfasserin |4 aut | |
| 700 | 1 | |a McBride, Marissa F |e verfasserin |4 aut | |
| 700 | 1 | |a Gould, Elliot |e verfasserin |4 aut | |
| 700 | 1 | |a Head, Andrew |e verfasserin |4 aut | |
| 700 | 1 | |a Hamilton, Daniel G |e verfasserin |4 aut | |
| 700 | 1 | |a Kambouris, Steven |e verfasserin |4 aut | |
| 700 | 1 | |a Rumpff, Libby |e verfasserin |4 aut | |
| 700 | 1 | |a Hoekstra, Rink |e verfasserin |4 aut | |
| 700 | 1 | |a Burgman, Mark A |e verfasserin |4 aut | |
| 700 | 1 | |a Fidler, Fiona |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t Royal Society open science |d 2014 |g 10(2023), 6 vom: 08. Juni, Seite 221553 |w (DE-627)NLM24986214X |x 2054-5703 |7 nnns |
| 773 | 1 | 8 | |g volume:10 |g year:2023 |g number:6 |g day:08 |g month:06 |g pages:221553 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1098/rsos.221553 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 10 |j 2023 |e 6 |b 08 |c 06 |h 221553 | ||