Automating risk of bias assessment in systematic reviews : a real-time mixed methods comparison of human researchers to a machine learning system
© 2022. The Author(s)..
BACKGROUND: Machine learning and automation are increasingly used to make the evidence synthesis process faster and more responsive to policymakers' needs. In systematic reviews of randomized controlled trials (RCTs), risk of bias assessment is a resource-intensive task that typically requires two trained reviewers. One function of RobotReviewer, an off-the-shelf machine learning system, is an automated risk of bias assessment.
METHODS: We assessed the feasibility of adopting RobotReviewer within a national public health institute using a randomized, real-time, user-centered study. The study included 26 RCTs and six reviewers from two projects examining health and social interventions. We randomized these studies to one of two RobotReviewer platforms. We operationalized feasibility as accuracy, time use, and reviewer acceptability. We measured accuracy by the number of corrections made by human reviewers (either to automated assessments or another human reviewer's assessments). We explored acceptability through group discussions and individual email responses after presenting the quantitative results.
RESULTS: Reviewers were equally likely to accept judgment by RobotReviewer as each other's judgement during the consensus process when measured dichotomously; risk ratio 1.02 (95% CI 0.92 to 1.13; p = 0.33). We were not able to compare time use. The acceptability of the program by researchers was mixed. Less experienced reviewers were generally more positive, and they saw more benefits and were able to use the tool more flexibly. Reviewers positioned human input and human-to-human interaction as superior to even a semi-automation of this process.
CONCLUSION: Despite being presented with evidence of RobotReviewer's equal performance to humans, participating reviewers were not interested in modifying standard procedures to include automation. If further studies confirm equal accuracy and reduced time compared to manual practices, we suggest that the benefits of RobotReviewer may support its future implementation as one of two assessors, despite reviewer ambivalence. Future research should study barriers to adopting automated tools and how highly educated and experienced researchers can adapt to a job market that is increasingly challenged by new technologies.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2022 |
|---|---|
| Erschienen: |
2022 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:22 |
|---|---|
| Enthalten in: |
BMC medical research methodology - 22(2022), 1 vom: 08. Juni, Seite 167 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Jardim, Patricia Sofia Jacobsen [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
Automation |
|---|
| Anmerkungen: |
Date Completed 10.06.2022 Date Revised 16.07.2022 published: Electronic Citation Status MEDLINE |
|---|
| doi: |
10.1186/s12874-022-01649-y |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM341983993 |
|---|
| LEADER | 01000naa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM341983993 | ||
| 003 | DE-627 | ||
| 005 | 20231226013028.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231226s2022 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1186/s12874-022-01649-y |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1139.xml |
| 035 | |a (DE-627)NLM341983993 | ||
| 035 | |a (NLM)35676632 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Jardim, Patricia Sofia Jacobsen |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Automating risk of bias assessment in systematic reviews |b a real-time mixed methods comparison of human researchers to a machine learning system |
| 264 | 1 | |c 2022 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 10.06.2022 | ||
| 500 | |a Date Revised 16.07.2022 | ||
| 500 | |a published: Electronic | ||
| 500 | |a Citation Status MEDLINE | ||
| 520 | |a © 2022. The Author(s). | ||
| 520 | |a BACKGROUND: Machine learning and automation are increasingly used to make the evidence synthesis process faster and more responsive to policymakers' needs. In systematic reviews of randomized controlled trials (RCTs), risk of bias assessment is a resource-intensive task that typically requires two trained reviewers. One function of RobotReviewer, an off-the-shelf machine learning system, is an automated risk of bias assessment | ||
| 520 | |a METHODS: We assessed the feasibility of adopting RobotReviewer within a national public health institute using a randomized, real-time, user-centered study. The study included 26 RCTs and six reviewers from two projects examining health and social interventions. We randomized these studies to one of two RobotReviewer platforms. We operationalized feasibility as accuracy, time use, and reviewer acceptability. We measured accuracy by the number of corrections made by human reviewers (either to automated assessments or another human reviewer's assessments). We explored acceptability through group discussions and individual email responses after presenting the quantitative results | ||
| 520 | |a RESULTS: Reviewers were equally likely to accept judgment by RobotReviewer as each other's judgement during the consensus process when measured dichotomously; risk ratio 1.02 (95% CI 0.92 to 1.13; p = 0.33). We were not able to compare time use. The acceptability of the program by researchers was mixed. Less experienced reviewers were generally more positive, and they saw more benefits and were able to use the tool more flexibly. Reviewers positioned human input and human-to-human interaction as superior to even a semi-automation of this process | ||
| 520 | |a CONCLUSION: Despite being presented with evidence of RobotReviewer's equal performance to humans, participating reviewers were not interested in modifying standard procedures to include automation. If further studies confirm equal accuracy and reduced time compared to manual practices, we suggest that the benefits of RobotReviewer may support its future implementation as one of two assessors, despite reviewer ambivalence. Future research should study barriers to adopting automated tools and how highly educated and experienced researchers can adapt to a job market that is increasingly challenged by new technologies | ||
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a Research Support, Non-U.S. Gov't | |
| 650 | 4 | |a Automation | |
| 650 | 4 | |a Evidence synthesis | |
| 650 | 4 | |a Heath technology assessment | |
| 650 | 4 | |a Machine learning | |
| 650 | 4 | |a Risk of bias | |
| 650 | 4 | |a RobotReviewer | |
| 650 | 4 | |a Systematic review | |
| 700 | 1 | |a Rose, Christopher James |e verfasserin |4 aut | |
| 700 | 1 | |a Ames, Heather Melanie |e verfasserin |4 aut | |
| 700 | 1 | |a Echavez, Jose Francisco Meneses |e verfasserin |4 aut | |
| 700 | 1 | |a Van de Velde, Stijn |e verfasserin |4 aut | |
| 700 | 1 | |a Muller, Ashley Elizabeth |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t BMC medical research methodology |d 2001 |g 22(2022), 1 vom: 08. Juni, Seite 167 |w (DE-627)NLM111431921 |x 1471-2288 |7 nnns |
| 773 | 1 | 8 | |g volume:22 |g year:2022 |g number:1 |g day:08 |g month:06 |g pages:167 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1186/s12874-022-01649-y |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 22 |j 2022 |e 1 |b 08 |c 06 |h 167 | ||