ICGA-GPT : report generation and question answering for indocyanine green angiography images
© Author(s) (or their employer(s)) 2024. No commercial re-use. See rights and permissions. Published by BMJ..
BACKGROUND: Indocyanine green angiography (ICGA) is vital for diagnosing chorioretinal diseases, but its interpretation and patient communication require extensive expertise and time-consuming efforts. We aim to develop a bilingual ICGA report generation and question-answering (QA) system.
METHODS: Our dataset comprised 213 129 ICGA images from 2919 participants. The system comprised two stages: image-text alignment for report generation by a multimodal transformer architecture, and large language model (LLM)-based QA with ICGA text reports and human-input questions. Performance was assessed using both qualitative metrics (including Bilingual Evaluation Understudy (BLEU), Consensus-based Image Description Evaluation (CIDEr), Recall-Oriented Understudy for Gisting Evaluation-Longest Common Subsequence (ROUGE-L), Semantic Propositional Image Caption Evaluation (SPICE), accuracy, sensitivity, specificity, precision and F1 score) and subjective evaluation by three experienced ophthalmologists using 5-point scales (5 refers to high quality).
RESULTS: We produced 8757 ICGA reports covering 39 disease-related conditions after bilingual translation (66.7% English, 33.3% Chinese). The ICGA-GPT model's report generation performance was evaluated with BLEU scores (1-4) of 0.48, 0.44, 0.40 and 0.37; CIDEr of 0.82; ROUGE of 0.41 and SPICE of 0.18. For disease-based metrics, the average specificity, accuracy, precision, sensitivity and F1 score were 0.98, 0.94, 0.70, 0.68 and 0.64, respectively. Assessing the quality of 50 images (100 reports), three ophthalmologists achieved substantial agreement (kappa=0.723 for completeness, kappa=0.738 for accuracy), yielding scores from 3.20 to 3.55. In an interactive QA scenario involving 100 generated answers, the ophthalmologists provided scores of 4.24, 4.22 and 4.10, displaying good consistency (kappa=0.779).
CONCLUSION: This pioneering study introduces the ICGA-GPT model for report generation and interactive QA for the first time, underscoring the potential of LLMs in assisting with automated ICGA image interpretation.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2024 |
|---|---|
| Erschienen: |
2024 |
| Enthalten in: |
Zur Gesamtaufnahme - year:2024 |
|---|---|
| Enthalten in: |
The British journal of ophthalmology - (2024) vom: 26. März |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Chen, Xiaolan [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
|---|
| Anmerkungen: |
Date Revised 26.03.2024 published: Print-Electronic Citation Status Publisher |
|---|
| doi: |
10.1136/bjo-2023-324446 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM369982541 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM369982541 | ||
| 003 | DE-627 | ||
| 005 | 20240328000245.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 240322s2024 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1136/bjo-2023-324446 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1351.xml |
| 035 | |a (DE-627)NLM369982541 | ||
| 035 | |a (NLM)38508675 | ||
| 035 | |a (PII)bjo-2023-324446 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Chen, Xiaolan |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a ICGA-GPT |b report generation and question answering for indocyanine green angiography images |
| 264 | 1 | |c 2024 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Revised 26.03.2024 | ||
| 500 | |a published: Print-Electronic | ||
| 500 | |a Citation Status Publisher | ||
| 520 | |a © Author(s) (or their employer(s)) 2024. No commercial re-use. See rights and permissions. Published by BMJ. | ||
| 520 | |a BACKGROUND: Indocyanine green angiography (ICGA) is vital for diagnosing chorioretinal diseases, but its interpretation and patient communication require extensive expertise and time-consuming efforts. We aim to develop a bilingual ICGA report generation and question-answering (QA) system | ||
| 520 | |a METHODS: Our dataset comprised 213 129 ICGA images from 2919 participants. The system comprised two stages: image-text alignment for report generation by a multimodal transformer architecture, and large language model (LLM)-based QA with ICGA text reports and human-input questions. Performance was assessed using both qualitative metrics (including Bilingual Evaluation Understudy (BLEU), Consensus-based Image Description Evaluation (CIDEr), Recall-Oriented Understudy for Gisting Evaluation-Longest Common Subsequence (ROUGE-L), Semantic Propositional Image Caption Evaluation (SPICE), accuracy, sensitivity, specificity, precision and F1 score) and subjective evaluation by three experienced ophthalmologists using 5-point scales (5 refers to high quality) | ||
| 520 | |a RESULTS: We produced 8757 ICGA reports covering 39 disease-related conditions after bilingual translation (66.7% English, 33.3% Chinese). The ICGA-GPT model's report generation performance was evaluated with BLEU scores (1-4) of 0.48, 0.44, 0.40 and 0.37; CIDEr of 0.82; ROUGE of 0.41 and SPICE of 0.18. For disease-based metrics, the average specificity, accuracy, precision, sensitivity and F1 score were 0.98, 0.94, 0.70, 0.68 and 0.64, respectively. Assessing the quality of 50 images (100 reports), three ophthalmologists achieved substantial agreement (kappa=0.723 for completeness, kappa=0.738 for accuracy), yielding scores from 3.20 to 3.55. In an interactive QA scenario involving 100 generated answers, the ophthalmologists provided scores of 4.24, 4.22 and 4.10, displaying good consistency (kappa=0.779) | ||
| 520 | |a CONCLUSION: This pioneering study introduces the ICGA-GPT model for report generation and interactive QA for the first time, underscoring the potential of LLMs in assisting with automated ICGA image interpretation | ||
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a Imaging | |
| 700 | 1 | |a Zhang, Weiyi |e verfasserin |4 aut | |
| 700 | 1 | |a Zhao, Ziwei |e verfasserin |4 aut | |
| 700 | 1 | |a Xu, Pusheng |e verfasserin |4 aut | |
| 700 | 1 | |a Zheng, Yingfeng |e verfasserin |4 aut | |
| 700 | 1 | |a Shi, Danli |e verfasserin |4 aut | |
| 700 | 1 | |a He, Mingguang |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t The British journal of ophthalmology |d 1917 |g (2024) vom: 26. März |w (DE-627)NLM000087556 |x 1468-2079 |7 nnns |
| 773 | 1 | 8 | |g year:2024 |g day:26 |g month:03 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1136/bjo-2023-324446 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |j 2024 |b 26 |c 03 | ||