Incremental Accumulation of Linguistic Context in Artificial and Biological Neural Networks
ABSTRACT Accumulated evidence suggests that Large Language Models (LLMs) are beneficial in predicting neural signals related to narrative processing. The way LLMs integrate context over large timescales, however, is fundamentally different from the way the brain does it. In this study, we show that unlike LLMs that apply parallel processing of large contextual windows, the incoming context to the brain is limited to short windows of a few tens of words. We hypothesize that whereas lower<jats:underline>-</jats:underline>level brain areas process short contextual windows, higher-order areas in the default-mode network (DMN) engage in an online incremental mechanism where the incoming short context is summarized and integrated with information accumulated across long timescales. Consequently, we introduce a novel LLM that instead of processing the entire context at once, it incrementally generates a concise summary of previous information. As predicted, we found that neural activities at the DMN were better predicted by the incremental model, and conversely, lower-level areas were better predicted with short-context-window LLM..
| Medienart: |
|
|---|
Preprint| Erscheinungsjahr: |
2024 |
|---|---|
| Erschienen: |
2024 |
| Enthalten in: |
bioRxiv.org - (2024) vom: 22. Jan. Zur Gesamtaufnahme - year:2024 |
|---|
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Tikochinski, Refael [VerfasserIn] |
|---|
| Links: |
Volltext [kostenfrei] |
|---|
| Themen: |
|---|
| doi: |
10.1101/2024.01.15.575798 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
XBI042179580 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | XBI042179580 | ||
| 003 | DE-627 | ||
| 005 | 20240123090417.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 240117s2024 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1101/2024.01.15.575798 |2 doi | |
| 035 | |a (DE-627)XBI042179580 | ||
| 035 | |a (biorXiv)10.1101/2024.01.15.575798 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Tikochinski, Refael |e verfasserin |0 (orcid)0000-0001-9092-8600 |4 aut | |
| 245 | 1 | 0 | |a Incremental Accumulation of Linguistic Context in Artificial and Biological Neural Networks |
| 264 | 1 | |c 2024 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a Computermedien |b c |2 rdamedia | ||
| 338 | |a Online-Ressource |b cr |2 rdacarrier | ||
| 520 | |a ABSTRACT Accumulated evidence suggests that Large Language Models (LLMs) are beneficial in predicting neural signals related to narrative processing. The way LLMs integrate context over large timescales, however, is fundamentally different from the way the brain does it. In this study, we show that unlike LLMs that apply parallel processing of large contextual windows, the incoming context to the brain is limited to short windows of a few tens of words. We hypothesize that whereas lower<jats:underline>-</jats:underline>level brain areas process short contextual windows, higher-order areas in the default-mode network (DMN) engage in an online incremental mechanism where the incoming short context is summarized and integrated with information accumulated across long timescales. Consequently, we introduce a novel LLM that instead of processing the entire context at once, it incrementally generates a concise summary of previous information. As predicted, we found that neural activities at the DMN were better predicted by the incremental model, and conversely, lower-level areas were better predicted with short-context-window LLM. | ||
| 650 | 4 | |a Biology |7 (dpeaa)DE-84 | |
| 650 | 4 | |a 570 |7 (dpeaa)DE-84 | |
| 700 | 1 | |a Goldstein, Ariel |0 (orcid)0000-0002-2393-1196 |4 aut | |
| 700 | 1 | |a Meiri, Yoav |4 aut | |
| 700 | 1 | |a Hasson, Uri |0 (orcid)0000-0002-3599-7168 |4 aut | |
| 700 | 1 | |a Reichart, Roi |0 (orcid)0000-0001-6918-0554 |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t bioRxiv.org |g (2024) vom: 22. Jan. |
| 773 | 1 | 8 | |g year:2024 |g day:22 |g month:01 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1101/2024.01.15.575798 |z kostenfrei |3 Volltext |
| 912 | |a GBV_XBI | ||
| 951 | |a AR | ||
| 952 | |j 2024 |b 22 |c 01 | ||