Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features
This work presents a novel method of exploring human brain-visual representations, with a view towards replicating these processes in machines. The core idea is to learn plausible computational and biological representations by correlating human neural activity and natural images. Thus, we first propose a model, EEG-ChannelNet, to learn a brain manifold for EEG classification. After verifying that visual information can be extracted from EEG data, we introduce a multimodal approach that uses deep image and EEG encoders, trained in a siamese configuration, for learning a joint manifold that maximizes a compatibility measure between visual features and brain representations. We then carry out image classification and saliency detection on the learned manifold. Performance analyses show that our approach satisfactorily decodes visual information from neural signals. This, in turn, can be used to effectively supervise the training of deep learning models, as demonstrated by the high performance of image classification and saliency detection on out-of-training classes. The obtained results show that the learned brain-visual features lead to improved performance and simultaneously bring deep models more in line with cognitive neuroscience work related to visual perception and attention.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2021 |
|---|---|
| Erschienen: |
2021 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:43 |
|---|---|
| Enthalten in: |
IEEE transactions on pattern analysis and machine intelligence - 43(2021), 11 vom: 30. Nov., Seite 3833-3849 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Palazzo, Simone [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
|---|
| Anmerkungen: |
Date Completed 10.12.2021 Date Revised 14.12.2021 published: Print-Electronic Citation Status MEDLINE |
|---|
| doi: |
10.1109/TPAMI.2020.2995909 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM313260079 |
|---|
| LEADER | 01000naa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM313260079 | ||
| 003 | DE-627 | ||
| 005 | 20231225150209.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231225s2021 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1109/TPAMI.2020.2995909 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1044.xml |
| 035 | |a (DE-627)NLM313260079 | ||
| 035 | |a (NLM)32750768 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Palazzo, Simone |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features |
| 264 | 1 | |c 2021 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 10.12.2021 | ||
| 500 | |a Date Revised 14.12.2021 | ||
| 500 | |a published: Print-Electronic | ||
| 500 | |a Citation Status MEDLINE | ||
| 520 | |a This work presents a novel method of exploring human brain-visual representations, with a view towards replicating these processes in machines. The core idea is to learn plausible computational and biological representations by correlating human neural activity and natural images. Thus, we first propose a model, EEG-ChannelNet, to learn a brain manifold for EEG classification. After verifying that visual information can be extracted from EEG data, we introduce a multimodal approach that uses deep image and EEG encoders, trained in a siamese configuration, for learning a joint manifold that maximizes a compatibility measure between visual features and brain representations. We then carry out image classification and saliency detection on the learned manifold. Performance analyses show that our approach satisfactorily decodes visual information from neural signals. This, in turn, can be used to effectively supervise the training of deep learning models, as demonstrated by the high performance of image classification and saliency detection on out-of-training classes. The obtained results show that the learned brain-visual features lead to improved performance and simultaneously bring deep models more in line with cognitive neuroscience work related to visual perception and attention | ||
| 650 | 4 | |a Journal Article | |
| 700 | 1 | |a Spampinato, Concetto |e verfasserin |4 aut | |
| 700 | 1 | |a Kavasidis, Isaak |e verfasserin |4 aut | |
| 700 | 1 | |a Giordano, Daniela |e verfasserin |4 aut | |
| 700 | 1 | |a Schmidt, Joseph |e verfasserin |4 aut | |
| 700 | 1 | |a Shah, Mubarak |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t IEEE transactions on pattern analysis and machine intelligence |d 1979 |g 43(2021), 11 vom: 30. Nov., Seite 3833-3849 |w (DE-627)NLM098212257 |x 1939-3539 |7 nnns |
| 773 | 1 | 8 | |g volume:43 |g year:2021 |g number:11 |g day:30 |g month:11 |g pages:3833-3849 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1109/TPAMI.2020.2995909 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 43 |j 2021 |e 11 |b 30 |c 11 |h 3833-3849 | ||