Accuracy and Efficiency of Machine Learning-Assisted Risk-of-Bias Assessments in "Real-World" Systematic Reviews : A Noninferiority Randomized Controlled Trial
BACKGROUND: Automation is a proposed solution for the increasing difficulty of maintaining up-to-date, high-quality health evidence. Evidence assessing the effectiveness of semiautomated data synthesis, such as risk-of-bias (RoB) assessments, is lacking.
OBJECTIVE: To determine whether RobotReviewer-assisted RoB assessments are noninferior in accuracy and efficiency to assessments conducted with human effort only.
DESIGN: Two-group, parallel, noninferiority, randomized trial. (Monash Research Office Project 11256).
SETTING: Health-focused systematic reviews using Covidence.
PARTICIPANTS: Systematic reviewers, who had not previously used RobotReviewer, completing Cochrane RoB assessments between February 2018 and May 2020.
INTERVENTION: In the intervention group, reviewers received an RoB form prepopulated by RobotReviewer; in the comparison group, reviewers received a blank form. Studies were assigned in a 1:1 ratio via simple randomization to receive RobotReviewer assistance for either Reviewer 1 or Reviewer 2. Participants were blinded to study allocation before starting work on each RoB form.
MEASUREMENTS: Co-primary outcomes were the accuracy of individual reviewer RoB assessments and the person-time required to complete individual assessments. Domain-level RoB accuracy was a secondary outcome.
RESULTS: Of the 15 recruited review teams, 7 completed the trial (145 included studies). Integration of RobotReviewer resulted in noninferior overall RoB assessment accuracy (risk difference, -0.014 [95% CI, -0.093 to 0.065]; intervention group: 88.8% accurate assessments; control group: 90.2% accurate assessments). Data were inconclusive for the person-time outcome (RobotReviewer saved 1.40 minutes [CI, -5.20 to 2.41 minutes]).
LIMITATION: Variability in user behavior and a limited number of assessable reviews led to an imprecise estimate of the time outcome.
CONCLUSION: In health-related systematic reviews, RoB assessments conducted with RobotReviewer assistance are noninferior in accuracy to those conducted without RobotReviewer assistance.
PRIMARY FUNDING SOURCE: University College London and Monash University.
| Errataetall: |
CommentIn: Ann Intern Med. 2022 Jul;175(7):1045-1046. - PMID 35635849 |
|---|---|
| Medienart: |
E-Artikel |
| Erscheinungsjahr: |
2022 |
|---|---|
| Erschienen: |
2022 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:175 |
|---|---|
| Enthalten in: |
Annals of internal medicine - 175(2022), 7 vom: 10. Juli, Seite 1001-1009 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Arno, Anneliese [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
Journal Article |
|---|
| Anmerkungen: |
Date Completed 20.07.2022 Date Revised 22.11.2022 published: Print-Electronic CommentIn: Ann Intern Med. 2022 Jul;175(7):1045-1046. - PMID 35635849 Citation Status MEDLINE |
|---|
| doi: |
10.7326/M22-0092 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM341581127 |
|---|
| LEADER | 01000naa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM341581127 | ||
| 003 | DE-627 | ||
| 005 | 20231226012118.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231226s2022 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.7326/M22-0092 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1138.xml |
| 035 | |a (DE-627)NLM341581127 | ||
| 035 | |a (NLM)35635850 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Arno, Anneliese |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Accuracy and Efficiency of Machine Learning-Assisted Risk-of-Bias Assessments in "Real-World" Systematic Reviews |b A Noninferiority Randomized Controlled Trial |
| 264 | 1 | |c 2022 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 20.07.2022 | ||
| 500 | |a Date Revised 22.11.2022 | ||
| 500 | |a published: Print-Electronic | ||
| 500 | |a CommentIn: Ann Intern Med. 2022 Jul;175(7):1045-1046. - PMID 35635849 | ||
| 500 | |a Citation Status MEDLINE | ||
| 520 | |a BACKGROUND: Automation is a proposed solution for the increasing difficulty of maintaining up-to-date, high-quality health evidence. Evidence assessing the effectiveness of semiautomated data synthesis, such as risk-of-bias (RoB) assessments, is lacking | ||
| 520 | |a OBJECTIVE: To determine whether RobotReviewer-assisted RoB assessments are noninferior in accuracy and efficiency to assessments conducted with human effort only | ||
| 520 | |a DESIGN: Two-group, parallel, noninferiority, randomized trial. (Monash Research Office Project 11256) | ||
| 520 | |a SETTING: Health-focused systematic reviews using Covidence | ||
| 520 | |a PARTICIPANTS: Systematic reviewers, who had not previously used RobotReviewer, completing Cochrane RoB assessments between February 2018 and May 2020 | ||
| 520 | |a INTERVENTION: In the intervention group, reviewers received an RoB form prepopulated by RobotReviewer; in the comparison group, reviewers received a blank form. Studies were assigned in a 1:1 ratio via simple randomization to receive RobotReviewer assistance for either Reviewer 1 or Reviewer 2. Participants were blinded to study allocation before starting work on each RoB form | ||
| 520 | |a MEASUREMENTS: Co-primary outcomes were the accuracy of individual reviewer RoB assessments and the person-time required to complete individual assessments. Domain-level RoB accuracy was a secondary outcome | ||
| 520 | |a RESULTS: Of the 15 recruited review teams, 7 completed the trial (145 included studies). Integration of RobotReviewer resulted in noninferior overall RoB assessment accuracy (risk difference, -0.014 [95% CI, -0.093 to 0.065]; intervention group: 88.8% accurate assessments; control group: 90.2% accurate assessments). Data were inconclusive for the person-time outcome (RobotReviewer saved 1.40 minutes [CI, -5.20 to 2.41 minutes]) | ||
| 520 | |a LIMITATION: Variability in user behavior and a limited number of assessable reviews led to an imprecise estimate of the time outcome | ||
| 520 | |a CONCLUSION: In health-related systematic reviews, RoB assessments conducted with RobotReviewer assistance are noninferior in accuracy to those conducted without RobotReviewer assistance | ||
| 520 | |a PRIMARY FUNDING SOURCE: University College London and Monash University | ||
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a Randomized Controlled Trial | |
| 650 | 4 | |a Research Support, Non-U.S. Gov't | |
| 700 | 1 | |a Thomas, James |e verfasserin |4 aut | |
| 700 | 1 | |a Wallace, Byron |e verfasserin |4 aut | |
| 700 | 1 | |a Marshall, Iain J |e verfasserin |4 aut | |
| 700 | 1 | |a McKenzie, Joanne E |e verfasserin |4 aut | |
| 700 | 1 | |a Elliott, Julian H |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t Annals of internal medicine |d 1945 |g 175(2022), 7 vom: 10. Juli, Seite 1001-1009 |w (DE-627)NLM000012777 |x 1539-3704 |7 nnns |
| 773 | 1 | 8 | |g volume:175 |g year:2022 |g number:7 |g day:10 |g month:07 |g pages:1001-1009 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.7326/M22-0092 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 175 |j 2022 |e 7 |b 10 |c 07 |h 1001-1009 | ||