A novel brain-computer interface based on audio-assisted visual evoked EEG and spatial-temporal attention CNN
Copyright © 2022 Chen, Zhang, Zhang, Li and Duan..
Objective: Brain-computer interface (BCI) can translate intentions directly into instructions and greatly improve the interaction experience for disabled people or some specific interactive applications. To improve the efficiency of BCI, the objective of this study is to explore the feasibility of an audio-assisted visual BCI speller and a deep learning-based single-trial event related potentials (ERP) decoding strategy.
Approach: In this study, a two-stage BCI speller combining the motion-onset visual evoked potential (mVEP) and semantically congruent audio evoked ERP was designed to output the target characters. In the first stage, the different group of characters were presented in the different locations of visual field simultaneously and the stimuli were coded to the mVEP based on a new space division multiple access scheme. And then, the target character can be output based on the audio-assisted mVEP in the second stage. Meanwhile, a spatial-temporal attention-based convolutional neural network (STA-CNN) was proposed to recognize the single-trial ERP components. The CNN can learn 2-dimentional features including the spatial information of different activated channels and time dependence among ERP components. In addition, the STA mechanism can enhance the discriminative event-related features by adaptively learning probability weights.
Main results: The performance of the proposed two-stage audio-assisted visual BCI paradigm and STA-CNN model was evaluated using the Electroencephalogram (EEG) recorded from 10 subjects. The average classification accuracy of proposed STA-CNN can reach 59.6 and 77.7% for the first and second stages, which were always significantly higher than those of the comparison methods (p < 0.05).
Significance: The proposed two-stage audio-assisted visual paradigm showed a great potential to be used to BCI speller. Moreover, through the analysis of the attention weights from time sequence and spatial topographies, it was proved that STA-CNN could effectively extract interpretable spatiotemporal EEG features.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2022 |
|---|---|
| Erschienen: |
2022 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:16 |
|---|---|
| Enthalten in: |
Frontiers in neurorobotics - 16(2022) vom: 25., Seite 995552 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Chen, Guijun [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
Audio-assisted visual evoked EEG |
|---|
| Anmerkungen: |
Date Revised 19.10.2022 published: Electronic-eCollection Citation Status PubMed-not-MEDLINE |
|---|
| doi: |
10.3389/fnbot.2022.995552 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM347623239 |
|---|
| LEADER | 01000naa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM347623239 | ||
| 003 | DE-627 | ||
| 005 | 20231226034253.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231226s2022 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.3389/fnbot.2022.995552 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1158.xml |
| 035 | |a (DE-627)NLM347623239 | ||
| 035 | |a (NLM)36247357 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Chen, Guijun |e verfasserin |4 aut | |
| 245 | 1 | 2 | |a A novel brain-computer interface based on audio-assisted visual evoked EEG and spatial-temporal attention CNN |
| 264 | 1 | |c 2022 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Revised 19.10.2022 | ||
| 500 | |a published: Electronic-eCollection | ||
| 500 | |a Citation Status PubMed-not-MEDLINE | ||
| 520 | |a Copyright © 2022 Chen, Zhang, Zhang, Li and Duan. | ||
| 520 | |a Objective: Brain-computer interface (BCI) can translate intentions directly into instructions and greatly improve the interaction experience for disabled people or some specific interactive applications. To improve the efficiency of BCI, the objective of this study is to explore the feasibility of an audio-assisted visual BCI speller and a deep learning-based single-trial event related potentials (ERP) decoding strategy | ||
| 520 | |a Approach: In this study, a two-stage BCI speller combining the motion-onset visual evoked potential (mVEP) and semantically congruent audio evoked ERP was designed to output the target characters. In the first stage, the different group of characters were presented in the different locations of visual field simultaneously and the stimuli were coded to the mVEP based on a new space division multiple access scheme. And then, the target character can be output based on the audio-assisted mVEP in the second stage. Meanwhile, a spatial-temporal attention-based convolutional neural network (STA-CNN) was proposed to recognize the single-trial ERP components. The CNN can learn 2-dimentional features including the spatial information of different activated channels and time dependence among ERP components. In addition, the STA mechanism can enhance the discriminative event-related features by adaptively learning probability weights | ||
| 520 | |a Main results: The performance of the proposed two-stage audio-assisted visual BCI paradigm and STA-CNN model was evaluated using the Electroencephalogram (EEG) recorded from 10 subjects. The average classification accuracy of proposed STA-CNN can reach 59.6 and 77.7% for the first and second stages, which were always significantly higher than those of the comparison methods (p < 0.05) | ||
| 520 | |a Significance: The proposed two-stage audio-assisted visual paradigm showed a great potential to be used to BCI speller. Moreover, through the analysis of the attention weights from time sequence and spatial topographies, it was proved that STA-CNN could effectively extract interpretable spatiotemporal EEG features | ||
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a audio-assisted visual evoked EEG | |
| 650 | 4 | |a brain-computer interface | |
| 650 | 4 | |a convolutional neural network | |
| 650 | 4 | |a space division multiple access | |
| 650 | 4 | |a spatial-temporal attention | |
| 700 | 1 | |a Zhang, Xueying |e verfasserin |4 aut | |
| 700 | 1 | |a Zhang, Jing |e verfasserin |4 aut | |
| 700 | 1 | |a Li, Fenglian |e verfasserin |4 aut | |
| 700 | 1 | |a Duan, Shufei |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t Frontiers in neurorobotics |d 2007 |g 16(2022) vom: 25., Seite 995552 |w (DE-627)NLM184032342 |x 1662-5218 |7 nnns |
| 773 | 1 | 8 | |g volume:16 |g year:2022 |g day:25 |g pages:995552 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.3389/fnbot.2022.995552 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 16 |j 2022 |b 25 |h 995552 | ||