Deep causal speech enhancement and recognition using efficient long-short term memory Recurrent Neural Network
Copyright: © 2024 Li et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited..
Long short-term memory (LSTM) has been effectively used to represent sequential data in recent years. However, LSTM still struggles with capturing the long-term temporal dependencies. In this paper, we propose an hourglass-shaped LSTM that is able to capture long-term temporal correlations by reducing the feature resolutions without data loss. We have used skip connections in non-adjacent layers to avoid gradient decay. In addition, an attention process is incorporated into skip connections to emphasize the essential spectral features and spectral regions. The proposed LSTM model is applied to speech enhancement and recognition applications. The proposed LSTM model uses no future information, resulting in a causal system suitable for real-time processing. The combined spectral feature sets are used to train the LSTM model for improved performance. Using the proposed model, the ideal ratio mask (IRM) is estimated as a training objective. The experimental evaluations using short-time objective intelligibility (STOI) and perceptual evaluation of speech quality (PESQ) have demonstrated that the proposed model with robust feature representation obtained higher speech intelligibility and perceptual quality. With the TIMIT, LibriSpeech, and VoiceBank datasets, the proposed model improved STOI by 16.21%, 16.41%, and 18.33% over noisy speech, whereas PESQ is improved by 31.1%, 32.9%, and 32%. In seen and unseen noisy situations, the proposed model outperformed existing deep neural networks (DNNs), including baseline LSTM, feedforward neural network (FDNN), convolutional neural network (CNN), and generative adversarial network (GAN). With the Kaldi toolkit for automated speech recognition (ASR), the proposed model significantly reduced the word error rates (WERs) and reached an average WER of 15.13% in noisy backgrounds.
Medienart: |
E-Artikel |
---|
Erscheinungsjahr: |
2024 |
---|---|
Erschienen: |
2024 |
Enthalten in: |
Zur Gesamtaufnahme - volume:19 |
---|---|
Enthalten in: |
PloS one - 19(2024), 1 vom: 31., Seite e0291240 |
Sprache: |
Englisch |
---|
Beteiligte Personen: |
Li, Zhenqing [VerfasserIn] |
---|
Links: |
---|
Themen: |
---|
Anmerkungen: |
Date Completed 05.01.2024 Date Revised 06.01.2024 published: Electronic-eCollection Citation Status MEDLINE |
---|
doi: |
10.1371/journal.pone.0291240 |
---|
funding: |
|
---|---|
Förderinstitution / Projekttitel: |
|
PPN (Katalog-ID): |
NLM366613308 |
---|
LEADER | 01000naa a22002652 4500 | ||
---|---|---|---|
001 | NLM366613308 | ||
003 | DE-627 | ||
005 | 20240108142639.0 | ||
007 | cr uuu---uuuuu | ||
008 | 240108s2024 xx |||||o 00| ||eng c | ||
024 | 7 | |a 10.1371/journal.pone.0291240 |2 doi | |
028 | 5 | 2 | |a pubmed24n1250.xml |
035 | |a (DE-627)NLM366613308 | ||
035 | |a (NLM)38170703 | ||
040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
041 | |a eng | ||
100 | 1 | |a Li, Zhenqing |e verfasserin |4 aut | |
245 | 1 | 0 | |a Deep causal speech enhancement and recognition using efficient long-short term memory Recurrent Neural Network |
264 | 1 | |c 2024 | |
336 | |a Text |b txt |2 rdacontent | ||
337 | |a ƒaComputermedien |b c |2 rdamedia | ||
338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
500 | |a Date Completed 05.01.2024 | ||
500 | |a Date Revised 06.01.2024 | ||
500 | |a published: Electronic-eCollection | ||
500 | |a Citation Status MEDLINE | ||
520 | |a Copyright: © 2024 Li et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. | ||
520 | |a Long short-term memory (LSTM) has been effectively used to represent sequential data in recent years. However, LSTM still struggles with capturing the long-term temporal dependencies. In this paper, we propose an hourglass-shaped LSTM that is able to capture long-term temporal correlations by reducing the feature resolutions without data loss. We have used skip connections in non-adjacent layers to avoid gradient decay. In addition, an attention process is incorporated into skip connections to emphasize the essential spectral features and spectral regions. The proposed LSTM model is applied to speech enhancement and recognition applications. The proposed LSTM model uses no future information, resulting in a causal system suitable for real-time processing. The combined spectral feature sets are used to train the LSTM model for improved performance. Using the proposed model, the ideal ratio mask (IRM) is estimated as a training objective. The experimental evaluations using short-time objective intelligibility (STOI) and perceptual evaluation of speech quality (PESQ) have demonstrated that the proposed model with robust feature representation obtained higher speech intelligibility and perceptual quality. With the TIMIT, LibriSpeech, and VoiceBank datasets, the proposed model improved STOI by 16.21%, 16.41%, and 18.33% over noisy speech, whereas PESQ is improved by 31.1%, 32.9%, and 32%. In seen and unseen noisy situations, the proposed model outperformed existing deep neural networks (DNNs), including baseline LSTM, feedforward neural network (FDNN), convolutional neural network (CNN), and generative adversarial network (GAN). With the Kaldi toolkit for automated speech recognition (ASR), the proposed model significantly reduced the word error rates (WERs) and reached an average WER of 15.13% in noisy backgrounds | ||
650 | 4 | |a Journal Article | |
700 | 1 | |a Basit, Abdul |e verfasserin |4 aut | |
700 | 1 | |a Daraz, Amil |e verfasserin |4 aut | |
700 | 1 | |a Jan, Atif |e verfasserin |4 aut | |
773 | 0 | 8 | |i Enthalten in |t PloS one |d 2006 |g 19(2024), 1 vom: 31., Seite e0291240 |w (DE-627)NLM167327399 |x 1932-6203 |7 nnns |
773 | 1 | 8 | |g volume:19 |g year:2024 |g number:1 |g day:31 |g pages:e0291240 |
856 | 4 | 0 | |u http://dx.doi.org/10.1371/journal.pone.0291240 |3 Volltext |
912 | |a GBV_USEFLAG_A | ||
912 | |a GBV_NLM | ||
951 | |a AR | ||
952 | |d 19 |j 2024 |e 1 |b 31 |h e0291240 |