Use of Large Language Models to Assess the Likelihood of Epidemics From the Content of Tweets : Infodemiology Study
©Michael S Deiner, Natalie A Deiner, Vagelis Hristidis, Stephen D McLeod, Thuy Doan, Thomas M Lietman, Travis C Porco. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 01.03.2024..
BACKGROUND: Previous work suggests that Google searches could be useful in identifying conjunctivitis epidemics. Content-based assessment of social media content may provide additional value in serving as early indicators of conjunctivitis and other systemic infectious diseases.
OBJECTIVE: We investigated whether large language models, specifically GPT-3.5 and GPT-4 (OpenAI), can provide probabilistic assessments of whether social media posts about conjunctivitis could indicate a regional outbreak.
METHODS: A total of 12,194 conjunctivitis-related tweets were obtained using a targeted Boolean search in multiple languages from India, Guam (United States), Martinique (France), the Philippines, American Samoa (United States), Fiji, Costa Rica, Haiti, and the Bahamas, covering the time frame from January 1, 2012, to March 13, 2023. By providing these tweets via prompts to GPT-3.5 and GPT-4, we obtained probabilistic assessments that were validated by 2 human raters. We then calculated Pearson correlations of these time series with tweet volume and the occurrence of known outbreaks in these 9 locations, with time series bootstrap used to compute CIs.
RESULTS: Probabilistic assessments derived from GPT-3.5 showed correlations of 0.60 (95% CI 0.47-0.70) and 0.53 (95% CI 0.40-0.65) with the 2 human raters, with higher results for GPT-4. The weekly averages of GPT-3.5 probabilities showed substantial correlations with weekly tweet volume for 44% (4/9) of the countries, with correlations ranging from 0.10 (95% CI 0.0-0.29) to 0.53 (95% CI 0.39-0.89), with larger correlations for GPT-4. More modest correlations were found for correlation with known epidemics, with substantial correlation only in American Samoa (0.40, 95% CI 0.16-0.81).
CONCLUSIONS: These findings suggest that GPT prompting can efficiently assess the content of social media posts and indicate possible disease outbreaks to a degree of accuracy comparable to that of humans. Furthermore, we found that automated content analysis of tweets is related to tweet volume for conjunctivitis-related posts in some locations and to the occurrence of actual epidemics. Future work may improve the sensitivity and specificity of these methods for disease outbreak detection.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2024 |
|---|---|
| Erschienen: |
2024 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:26 |
|---|---|
| Enthalten in: |
Journal of medical Internet research - 26(2024) vom: 01. März, Seite e49139 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Deiner, Michael S [VerfasserIn] |
|---|
| Links: |
|---|
| Anmerkungen: |
Date Completed 04.03.2024 Date Revised 18.03.2024 published: Electronic Citation Status MEDLINE |
|---|
| doi: |
10.2196/49139 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM369172930 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM369172930 | ||
| 003 | DE-627 | ||
| 005 | 20240318234539.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 240301s2024 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.2196/49139 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1334.xml |
| 035 | |a (DE-627)NLM369172930 | ||
| 035 | |a (NLM)38427404 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Deiner, Michael S |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Use of Large Language Models to Assess the Likelihood of Epidemics From the Content of Tweets |b Infodemiology Study |
| 264 | 1 | |c 2024 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 04.03.2024 | ||
| 500 | |a Date Revised 18.03.2024 | ||
| 500 | |a published: Electronic | ||
| 500 | |a Citation Status MEDLINE | ||
| 520 | |a ©Michael S Deiner, Natalie A Deiner, Vagelis Hristidis, Stephen D McLeod, Thuy Doan, Thomas M Lietman, Travis C Porco. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 01.03.2024. | ||
| 520 | |a BACKGROUND: Previous work suggests that Google searches could be useful in identifying conjunctivitis epidemics. Content-based assessment of social media content may provide additional value in serving as early indicators of conjunctivitis and other systemic infectious diseases | ||
| 520 | |a OBJECTIVE: We investigated whether large language models, specifically GPT-3.5 and GPT-4 (OpenAI), can provide probabilistic assessments of whether social media posts about conjunctivitis could indicate a regional outbreak | ||
| 520 | |a METHODS: A total of 12,194 conjunctivitis-related tweets were obtained using a targeted Boolean search in multiple languages from India, Guam (United States), Martinique (France), the Philippines, American Samoa (United States), Fiji, Costa Rica, Haiti, and the Bahamas, covering the time frame from January 1, 2012, to March 13, 2023. By providing these tweets via prompts to GPT-3.5 and GPT-4, we obtained probabilistic assessments that were validated by 2 human raters. We then calculated Pearson correlations of these time series with tweet volume and the occurrence of known outbreaks in these 9 locations, with time series bootstrap used to compute CIs | ||
| 520 | |a RESULTS: Probabilistic assessments derived from GPT-3.5 showed correlations of 0.60 (95% CI 0.47-0.70) and 0.53 (95% CI 0.40-0.65) with the 2 human raters, with higher results for GPT-4. The weekly averages of GPT-3.5 probabilities showed substantial correlations with weekly tweet volume for 44% (4/9) of the countries, with correlations ranging from 0.10 (95% CI 0.0-0.29) to 0.53 (95% CI 0.39-0.89), with larger correlations for GPT-4. More modest correlations were found for correlation with known epidemics, with substantial correlation only in American Samoa (0.40, 95% CI 0.16-0.81) | ||
| 520 | |a CONCLUSIONS: These findings suggest that GPT prompting can efficiently assess the content of social media posts and indicate possible disease outbreaks to a degree of accuracy comparable to that of humans. Furthermore, we found that automated content analysis of tweets is related to tweet volume for conjunctivitis-related posts in some locations and to the occurrence of actual epidemics. Future work may improve the sensitivity and specificity of these methods for disease outbreak detection | ||
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a GPT-3.5 | |
| 650 | 4 | |a GPT-4 | |
| 650 | 4 | |a Generative Pre-trained Transformers | |
| 650 | 4 | |a Twitter | |
| 650 | 4 | |a X formerly known as Twitter | |
| 650 | 4 | |a conjunctivitis | |
| 650 | 4 | |a epidemic detection | |
| 650 | 4 | |a generative large language model | |
| 650 | 4 | |a infectious eye disease | |
| 650 | 4 | |a microblog | |
| 650 | 4 | |a social media | |
| 700 | 1 | |a Deiner, Natalie A |e verfasserin |4 aut | |
| 700 | 1 | |a Hristidis, Vagelis |e verfasserin |4 aut | |
| 700 | 1 | |a McLeod, Stephen D |e verfasserin |4 aut | |
| 700 | 1 | |a Doan, Thuy |e verfasserin |4 aut | |
| 700 | 1 | |a Lietman, Thomas M |e verfasserin |4 aut | |
| 700 | 1 | |a Porco, Travis C |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t Journal of medical Internet research |d 1999 |g 26(2024) vom: 01. März, Seite e49139 |w (DE-627)NLM116127104 |x 1438-8871 |7 nnns |
| 773 | 1 | 8 | |g volume:26 |g year:2024 |g day:01 |g month:03 |g pages:e49139 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.2196/49139 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 26 |j 2024 |b 01 |c 03 |h e49139 | ||