Brain-informed speech separation (BISS) for enhancement of target speaker in multitalker speech perception
Copyright © 2020. Published by Elsevier Inc..
Hearing-impaired people often struggle to follow the speech stream of an individual talker in noisy environments. Recent studies show that the brain tracks attended speech and that the attended talker can be decoded from neural data on a single-trial level. This raises the possibility of "neuro-steered" hearing devices in which the brain-decoded intention of a hearing-impaired listener is used to enhance the voice of the attended speaker from a speech separation front-end. So far, methods that use this paradigm have focused on optimizing the brain decoding and the acoustic speech separation independently. In this work, we propose a novel framework called brain-informed speech separation (BISS)1 in which the information about the attended speech, as decoded from the subject's brain, is directly used to perform speech separation in the front-end. We present a deep learning model that uses neural data to extract the clean audio signal that a listener is attending to from a multi-talker speech mixture. We show that the framework can be applied successfully to the decoded output from either invasive intracranial electroencephalography (iEEG) or non-invasive electroencephalography (EEG) recordings from hearing-impaired subjects. It also results in improved speech separation, even in scenes with background noise. The generalization capability of the system renders it a perfect candidate for neuro-steered hearing-assistive devices.
Medienart: |
E-Artikel |
---|
Erscheinungsjahr: |
2020 |
---|---|
Erschienen: |
2020 |
Enthalten in: |
Zur Gesamtaufnahme - volume:223 |
---|---|
Enthalten in: |
NeuroImage - 223(2020) vom: 15. Dez., Seite 117282 |
Sprache: |
Englisch |
---|
Beteiligte Personen: |
Ceolini, Enea [VerfasserIn] |
---|
Links: |
---|
Anmerkungen: |
Date Completed 03.03.2021 Date Revised 31.07.2021 published: Print-Electronic Citation Status MEDLINE |
---|
doi: |
10.1016/j.neuroimage.2020.117282 |
---|
funding: |
|
---|---|
Förderinstitution / Projekttitel: |
|
PPN (Katalog-ID): |
NLM314025553 |
---|
LEADER | 01000naa a22002652 4500 | ||
---|---|---|---|
001 | NLM314025553 | ||
003 | DE-627 | ||
005 | 20231225151836.0 | ||
007 | cr uuu---uuuuu | ||
008 | 231225s2020 xx |||||o 00| ||eng c | ||
024 | 7 | |a 10.1016/j.neuroimage.2020.117282 |2 doi | |
028 | 5 | 2 | |a pubmed24n1046.xml |
035 | |a (DE-627)NLM314025553 | ||
035 | |a (NLM)32828921 | ||
035 | |a (PII)S1053-8119(20)30768-0 | ||
040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
041 | |a eng | ||
100 | 1 | |a Ceolini, Enea |e verfasserin |4 aut | |
245 | 1 | 0 | |a Brain-informed speech separation (BISS) for enhancement of target speaker in multitalker speech perception |
264 | 1 | |c 2020 | |
336 | |a Text |b txt |2 rdacontent | ||
337 | |a ƒaComputermedien |b c |2 rdamedia | ||
338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
500 | |a Date Completed 03.03.2021 | ||
500 | |a Date Revised 31.07.2021 | ||
500 | |a published: Print-Electronic | ||
500 | |a Citation Status MEDLINE | ||
520 | |a Copyright © 2020. Published by Elsevier Inc. | ||
520 | |a Hearing-impaired people often struggle to follow the speech stream of an individual talker in noisy environments. Recent studies show that the brain tracks attended speech and that the attended talker can be decoded from neural data on a single-trial level. This raises the possibility of "neuro-steered" hearing devices in which the brain-decoded intention of a hearing-impaired listener is used to enhance the voice of the attended speaker from a speech separation front-end. So far, methods that use this paradigm have focused on optimizing the brain decoding and the acoustic speech separation independently. In this work, we propose a novel framework called brain-informed speech separation (BISS)1 in which the information about the attended speech, as decoded from the subject's brain, is directly used to perform speech separation in the front-end. We present a deep learning model that uses neural data to extract the clean audio signal that a listener is attending to from a multi-talker speech mixture. We show that the framework can be applied successfully to the decoded output from either invasive intracranial electroencephalography (iEEG) or non-invasive electroencephalography (EEG) recordings from hearing-impaired subjects. It also results in improved speech separation, even in scenes with background noise. The generalization capability of the system renders it a perfect candidate for neuro-steered hearing-assistive devices | ||
650 | 4 | |a Journal Article | |
650 | 4 | |a Research Support, N.I.H., Extramural | |
650 | 4 | |a Research Support, Non-U.S. Gov't | |
650 | 4 | |a Research Support, U.S. Gov't, Non-P.H.S. | |
650 | 4 | |a Cognitive control | |
650 | 4 | |a Deep learning | |
650 | 4 | |a EEG | |
650 | 4 | |a Hearing aid | |
650 | 4 | |a Neuro-steered | |
650 | 4 | |a Speech separation | |
700 | 1 | |a Hjortkjær, Jens |e verfasserin |4 aut | |
700 | 1 | |a Wong, Daniel D E |e verfasserin |4 aut | |
700 | 1 | |a O'Sullivan, James |e verfasserin |4 aut | |
700 | 1 | |a Raghavan, Vinay S |e verfasserin |4 aut | |
700 | 1 | |a Herrero, Jose |e verfasserin |4 aut | |
700 | 1 | |a Mehta, Ashesh D |e verfasserin |4 aut | |
700 | 1 | |a Liu, Shih-Chii |e verfasserin |4 aut | |
700 | 1 | |a Mesgarani, Nima |e verfasserin |4 aut | |
773 | 0 | 8 | |i Enthalten in |t NeuroImage |d 1992 |g 223(2020) vom: 15. Dez., Seite 117282 |w (DE-627)NLM09001443X |x 1095-9572 |7 nnns |
773 | 1 | 8 | |g volume:223 |g year:2020 |g day:15 |g month:12 |g pages:117282 |
856 | 4 | 0 | |u http://dx.doi.org/10.1016/j.neuroimage.2020.117282 |3 Volltext |
912 | |a GBV_USEFLAG_A | ||
912 | |a GBV_NLM | ||
951 | |a AR | ||
952 | |d 223 |j 2020 |b 15 |c 12 |h 117282 |