Development of meta-prompts for Large Language Models to screen titles and abstracts for diagnostic test accuracy reviews
Abstract Systematic reviews (SRs) are a critical component of evidence-based medicine, but the process of screening titles and abstracts is time-consuming. This study aimed to develop and externally validate a method using large language models to classify abstracts for diagnostic test accuracy (DTA) systematic reviews, thereby reducing the human workload. We used a previously collected dataset for developing DTA abstract classifiers and applied prompt engineering. We developed an optimized meta-prompt for Generative Pre-trained Transformer (GPT)-3.5-turbo and GPT-4 to classify abstracts. In the external validation dataset 1, the prompt with GPT-3.5 turbo showed a sensitivity of 0.988, and a specificity of 0.298. GPT-4 showed a sensitivity of 0.982, and a specificity of 0.677. In the external validation dataset 2, GPT-3.5 turbo showed a sensitivity of 0.919, and a specificity of 0.434. GPT-4 showed a sensitivity of 0.806, and a specificity of 0.740. If we included eligible studies from among the references of the identified studies, GPT-3.5 turbo had no critical misses, while GPT-4 had some misses. Our study indicates that GPT-3.5 turbo can be effectively used to classify abstracts for DTA systematic reviews. Further studies using other dataset are warranted to confirm our results. Additionally, we encourage the use of our framework and publicly available dataset for further exploration of more effective classifiers using other LLMs and prompts (<jats:ext-link xmlns:xlink="http://www.w3.org/1999/xlink" ext-link-type="uri" xlink:href="https://github.com/youkiti/ARE/">https://github.com/youkiti/ARE/</jats:ext-link>).Hightlights What is already known <jats:list list-type="simple">- Title and abstract screening in systematic reviews (SRs) consumes significant time.- Several attempts using machine learning to reduce this process in diagnostic test accuracy (DTA) SRs exist, but they have not yielded positive results in external validation.What is new <jats:list list-type="simple">- We aimed to develop and externally validate optimized meta-prompt for GPT-3.5-turbo and GPT-4 to classify abstracts for DTA SRs.- Through an iterative approach across three training datasets, an optimal meta-prompt capable of identifying DTA studies with remarkable sensitivity and specificity was developed.- The accuracy reproduced in the external validation datasets.Potential Impact for Readers <jats:list list-type="simple">- The developed meta-prompt can lessen the need for humans to read abstracts for DTA SRs, saving significant time and resources..
| Medienart: |
|
|---|
Preprint| Erscheinungsjahr: |
2023 |
|---|---|
| Erschienen: |
2023 |
| Enthalten in: |
bioRxiv.org - (2023) vom: 03. Nov. Zur Gesamtaufnahme - year:2023 |
|---|
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Kataoka, Yuki [VerfasserIn] |
|---|
| Links: |
Volltext [kostenfrei] |
|---|
| Themen: |
|---|
| doi: |
10.1101/2023.10.31.23297818 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
XBI041401301 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | XBI041401301 | ||
| 003 | DE-627 | ||
| 005 | 20231205143810.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231102s2023 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1101/2023.10.31.23297818 |2 doi | |
| 035 | |a (DE-627)XBI041401301 | ||
| 035 | |a (biorXiv)10.1101/2023.10.31.23297818 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Kataoka, Yuki |e verfasserin |0 (orcid)0000-0001-7982-5213 |4 aut | |
| 245 | 1 | 0 | |a Development of meta-prompts for Large Language Models to screen titles and abstracts for diagnostic test accuracy reviews |
| 264 | 1 | |c 2023 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a Computermedien |b c |2 rdamedia | ||
| 338 | |a Online-Ressource |b cr |2 rdacarrier | ||
| 520 | |a Abstract Systematic reviews (SRs) are a critical component of evidence-based medicine, but the process of screening titles and abstracts is time-consuming. This study aimed to develop and externally validate a method using large language models to classify abstracts for diagnostic test accuracy (DTA) systematic reviews, thereby reducing the human workload. We used a previously collected dataset for developing DTA abstract classifiers and applied prompt engineering. We developed an optimized meta-prompt for Generative Pre-trained Transformer (GPT)-3.5-turbo and GPT-4 to classify abstracts. In the external validation dataset 1, the prompt with GPT-3.5 turbo showed a sensitivity of 0.988, and a specificity of 0.298. GPT-4 showed a sensitivity of 0.982, and a specificity of 0.677. In the external validation dataset 2, GPT-3.5 turbo showed a sensitivity of 0.919, and a specificity of 0.434. GPT-4 showed a sensitivity of 0.806, and a specificity of 0.740. If we included eligible studies from among the references of the identified studies, GPT-3.5 turbo had no critical misses, while GPT-4 had some misses. Our study indicates that GPT-3.5 turbo can be effectively used to classify abstracts for DTA systematic reviews. Further studies using other dataset are warranted to confirm our results. Additionally, we encourage the use of our framework and publicly available dataset for further exploration of more effective classifiers using other LLMs and prompts (<jats:ext-link xmlns:xlink="http://www.w3.org/1999/xlink" ext-link-type="uri" xlink:href="https://github.com/youkiti/ARE/">https://github.com/youkiti/ARE/</jats:ext-link>).Hightlights What is already known <jats:list list-type="simple">- Title and abstract screening in systematic reviews (SRs) consumes significant time.- Several attempts using machine learning to reduce this process in diagnostic test accuracy (DTA) SRs exist, but they have not yielded positive results in external validation.What is new <jats:list list-type="simple">- We aimed to develop and externally validate optimized meta-prompt for GPT-3.5-turbo and GPT-4 to classify abstracts for DTA SRs.- Through an iterative approach across three training datasets, an optimal meta-prompt capable of identifying DTA studies with remarkable sensitivity and specificity was developed.- The accuracy reproduced in the external validation datasets.Potential Impact for Readers <jats:list list-type="simple">- The developed meta-prompt can lessen the need for humans to read abstracts for DTA SRs, saving significant time and resources. | ||
| 650 | 4 | |a Biology |7 (dpeaa)DE-84 | |
| 650 | 4 | |a 570 |7 (dpeaa)DE-84 | |
| 700 | 1 | |a So, Ryuhei |0 (orcid)0000-0002-9838-350X |4 aut | |
| 700 | 1 | |a Banno, Masahiro |0 (orcid)0000-0002-2539-1031 |4 aut | |
| 700 | 1 | |a Kumasawa, Junji |0 (orcid)0000-0003-4619-945X |4 aut | |
| 700 | 1 | |a Someko, Hidehiro |0 (orcid)0000-0002-7195-2055 |4 aut | |
| 700 | 1 | |a Taito, Shunsuke |0 (orcid)0000-0003-1218-4225 |4 aut | |
| 700 | 1 | |a Terasawa, Teruhiko |4 aut | |
| 700 | 1 | |a Tsujimoto, Yasushi |0 (orcid)0000-0002-7214-5589 |4 aut | |
| 700 | 1 | |a Tsutsumi, Yusuke |0 (orcid)0000-0002-9160-0241 |4 aut | |
| 700 | 1 | |a Wada, Yoshitaka |0 (orcid)0000-0003-2191-3629 |4 aut | |
| 700 | 1 | |a Furukawa, Toshi A. |0 (orcid)0000-0003-2159-3776 |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t bioRxiv.org |g (2023) vom: 03. Nov. |
| 773 | 1 | 8 | |g year:2023 |g day:03 |g month:11 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1101/2023.10.31.23297818 |z kostenfrei |3 Volltext |
| 912 | |a GBV_XBI | ||
| 951 | |a AR | ||
| 952 | |j 2023 |b 03 |c 11 | ||