Shared functional specialization in transformer-based language models and the human brain
Abstract Humans use complex linguistic structures to transmit ideas to one another. The brain is thought to deploy specialized computations to process these structures. Recently, a new class of artificial neural networks based on the Transformer architecture has revolutionized the field of language modeling, attracting attention from neuroscientists seeking to understand the neurobiology of languagein silico. Transformers integrate information across words via multiple layers of structured circuit computations, forming increasingly contextualized representations of linguistic content. Prior work has focused on the internal representations (the “embeddings”) generated by these circuits. In this paper, we instead analyze the circuit computations directly: we deconstruct these computations into functionally-specialized “transformations” to provide a complementary window onto linguistic computations in the human brain. Using functional MRI data acquired while participants listened to naturalistic spoken stories, we first verify that the transformations account for considerable variance in brain activity across the cortical language network. We then demonstrate that the emergent syntactic computations performed by individual, functionally-specialized “attention heads” differentially predict brain activity in specific cortical regions. These heads fall along gradients corresponding to different layers, contextual distances, and syntactic dependencies in a low-dimensional cortical space. Our findings indicate that large language models and the cortical language network may converge on similar trends of functional specialization for processing natural language..
| Medienart: |
|
|---|
Preprint| Erscheinungsjahr: |
2023 |
|---|---|
| Erschienen: |
2023 |
| Enthalten in: |
bioRxiv.org - (2023) vom: 25. Juli Zur Gesamtaufnahme - year:2023 |
|---|
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Kumar, Sreejan [VerfasserIn] |
|---|
| Links: |
Volltext [kostenfrei] |
|---|
| Themen: |
|---|
| doi: |
10.1101/2022.06.08.495348 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
XBI037508490 |
|---|
| LEADER | 01000caa a22002652 4500 | ||
|---|---|---|---|
| 001 | XBI037508490 | ||
| 003 | DE-627 | ||
| 005 | 20231205145926.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 221008s2023 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1101/2022.06.08.495348 |2 doi | |
| 035 | |a (DE-627)XBI037508490 | ||
| 035 | |a (biorXiv)10.1101/2022.06.08.495348 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Kumar, Sreejan |e verfasserin |0 (orcid)0000-0003-1769-5147 |4 aut | |
| 245 | 1 | 0 | |a Shared functional specialization in transformer-based language models and the human brain |
| 264 | 1 | |c 2023 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a Computermedien |b c |2 rdamedia | ||
| 338 | |a Online-Ressource |b cr |2 rdacarrier | ||
| 520 | |a Abstract Humans use complex linguistic structures to transmit ideas to one another. The brain is thought to deploy specialized computations to process these structures. Recently, a new class of artificial neural networks based on the Transformer architecture has revolutionized the field of language modeling, attracting attention from neuroscientists seeking to understand the neurobiology of languagein silico. Transformers integrate information across words via multiple layers of structured circuit computations, forming increasingly contextualized representations of linguistic content. Prior work has focused on the internal representations (the “embeddings”) generated by these circuits. In this paper, we instead analyze the circuit computations directly: we deconstruct these computations into functionally-specialized “transformations” to provide a complementary window onto linguistic computations in the human brain. Using functional MRI data acquired while participants listened to naturalistic spoken stories, we first verify that the transformations account for considerable variance in brain activity across the cortical language network. We then demonstrate that the emergent syntactic computations performed by individual, functionally-specialized “attention heads” differentially predict brain activity in specific cortical regions. These heads fall along gradients corresponding to different layers, contextual distances, and syntactic dependencies in a low-dimensional cortical space. Our findings indicate that large language models and the cortical language network may converge on similar trends of functional specialization for processing natural language. | ||
| 650 | 4 | |a Biology |7 (dpeaa)DE-84 | |
| 650 | 4 | |a 570 |7 (dpeaa)DE-84 | |
| 700 | 1 | |a Sumers, Theodore R. |0 (orcid)0000-0002-6128-0291 |4 aut | |
| 700 | 1 | |a Yamakoshi, Takateru |0 (orcid)0000-0003-2190-2524 |4 aut | |
| 700 | 1 | |a Goldstein, Ariel |0 (orcid)0000-0002-2393-1196 |4 aut | |
| 700 | 1 | |a Hasson, Uri |0 (orcid)0000-0002-3599-7168 |4 aut | |
| 700 | 1 | |a Norman, Kenneth A. |0 (orcid)0000-0002-5887-9682 |4 aut | |
| 700 | 1 | |a Griffiths, Thomas L. |0 (orcid)0000-0002-5138-7255 |4 aut | |
| 700 | 1 | |a Hawkins, Robert D. |0 (orcid)0000-0001-9089-8544 |4 aut | |
| 700 | 1 | |a Nastase, Samuel A. |0 (orcid)0000-0001-7013-5275 |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t bioRxiv.org |g (2023) vom: 25. Juli |
| 773 | 1 | 8 | |g year:2023 |g day:25 |g month:07 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1101/2022.06.08.495348 |z kostenfrei |3 Volltext |
| 912 | |a GBV_XBI | ||
| 951 | |a AR | ||
| 952 | |j 2023 |b 25 |c 07 | ||