Cochlear implant users struggle to understand speech in reverberant environments. To restore speech perception, artifacts dominated by reverberant reflections can be removed from the cochlear implant stimulus. Artifacts can be identified and removed by applying a matrix of gain values, a technique referred to as time-frequency masking. Gain values are determined by an oracle algorithm that uses knowledge of the undistorted signal to minimize retention of the signal components dominated by reverberant reflections. In practice, gain values are estimated from the distorted signal, with the oracle algorithm providing the estimation objective. Different oracle techniques exist for determining gain values, and each technique must be parameterized to set the amount of signal retention. This work assesses which oracle masking strategies and parameterizations lead to the best improvements in speech intelligibility for cochlear implant users in reverberant conditions using online speech intelligibility testing of normal-hearing individuals with vocoding.
Cochlear implant (CI) users have considerable difficulty in understanding speech in reverberant listening environments. Time-frequency (T-F) masking is a common technique that aims to improve speech intelligibility by multiplying reverberant speech by a matrix of gain values to suppress T-F bins dominated by reverberation. Recently proposed mask estimation algorithms leverage machine learning approaches to distinguish between target speech and reverberant reflections. However, the spectro-temporal structure of speech is highly variable and dependent on the underlying phoneme. One way to potent…
Enhancing spatial hearing with cochlear implants: exploring the role of AI, multimodal interaction and perceptual training
Acoustics Research Institute, Austrian Academy of Sciences（奥地利科学院声学研究所）；Centre de Recherche en Neurosciences de Lyon Inserm（里昂神经科学研究中心（Inserm））；Department of Neural Dynamics and Magnetoencephalography, Hertie Institute for Clinical Brain Research, University of Tübingen（图宾根大学神经动力学与脑磁图部门）；Centre for Integrative Neuroscience, University of Tübingen（图宾根大学整合神经科学中心）；NEMO Labs Nonprofit Kft., Budapest（布达佩斯NEMO实验室（非营利公司））
Research Center for Information Technology Innovation, Academia Sinica（中央研究院资讯科技创新研究中心）；National Taiwan University（台湾大学）；Institute of Linguistics, Academia Sinica（中央研究院语言学研究所）
Speech perception is key to verbal communication. For people with hearing loss, the capability to recognize speech is restricted, particularly in a noisy environment or the situations without visual cues, such as lip-reading unavailable via phone call. This study aimed to understand the improvement of vocoded speech intelligibility in cochlear implant (CI) simulation through two potential methods: Speech Enhancement (SE) and Audiovisual Integration. A fully convolutional neural network (FCN) using an intelligibility-oriented objective function was recently proposed and proven to effectively fa…
DeepSpeech models show Human-like Performance and Processing of Cochlear Implant Inputs
Center for Theoretical Neuroscience（理论神经科学中心）；Zuckerman Mind Brain Behavior Institute（扎克曼脑与行为研究所）；Columbia University（哥伦比亚大学）；Google DeepMind（谷歌DeepMind）
Speaker recognition is a biometric modality that uses underlying speech information to determine the identity of the speaker. Speaker Identification (SID) under noisy conditions is one of the challenging topics in the field of speech processing, specifically when it comes to individuals with cochlear implants (CI). This study analyzes and quantifies the ability of CI-users to perform speaker identification based on direct electric auditory stimuli. CI users employ a limited number of frequency bands (8 to 22) and use electrodes to directly stimulate the Basilar Membrane/Cochlear in order to re…
R.V. College of Engineering（R.V.工程学院）；JNT University（JNT大学）
Cochlear implant devices are known to exist since a long time. The purpose of the present work is to develop a speech algorithm for obtaining robust speech. In this paper, the technique of cochlear implant is first introduced, followed by discussions of some of the existing techniques available for obtaining speech. The next section introduces a new technique for obtaining robust speech. The key feature of this technique lies in the use of the advantages of an integrated approach involving the use of an estimation technique such as a kalman filter with non linear filter bank strategy, using Du…
Attempts to develop speech enhancement algorithms with improved speech intelligibility for cochlear implant (CI) users have met with limited success. To improve speech enhancement methods for CI users, we propose to perform speech enhancement in a cochlear filter-bank feature space, a feature-set specifically designed for CI users based on CI auditory stimuli. We leverage a convolutional neural network (CNN) to extract both stationary and non-stationary components of environmental acoustics and speech. We propose three CNN architectures: (1) vanilla CNN that directly generates the enhanced sig…
This dissertation covers a single-processor approach to the speech processing pipeline of bilateral Cochlear Implants (CIs). The use of only a single processor to provide binaural stimulation signals overcomes the synchronization problem, which is an existing challenging problem in the deployment of bilateral CI devices. The developed single-processor speech processing pipeline provides CI users with a sense of directionality. Its non-synchronization feature as well as low computational and memory requirements make it a suitable solution for actual deployment. A speech enhancement framework is…
Pruning-aware Loss Functions for STOI-Optimized Pruned Recurrent Autoencoders for the Compression of the Stimulation Patterns of Cochlear Implants at Zero Delay
End-to-end audio-visual learning for cochlear implant sound coding simulations in noisy environments
Graduate Institute of Electronics Engineering（电子工程研究所）；the Department of Electrical Engineering, National Taiwan University, Taipei 106319, Taiwan（国立台湾大学电子工程系）；Research Center for Information Technology Innovation Technology, Academia Sinica, Taipei 115201, Taiwan（中科院资讯科技创新研究中心）；Department of Electrical Engineering, Chung Yuan Christian University, Taoyuan 320314, Taiwan（Chung Yuan Christian University 电子工程系）
University of Texas at Dallas（德克萨斯大学达拉斯分校）；Tongmyong University（东明大学）
In multi speakers environments, cochlear implant (CI) users may attend to a target sound source in a different manner from the normal hearing (NH) individuals during a conversation. This study attempted to investigate the effect of conversational sound levels on the mechanisms adopted by CI and NH listeners in selective auditory attention and how it affects their daily conversation. Nine CI users (five bilateral, three unilateral, and one bimodal) and eight NH listeners participated in this study. The behavioral speech recognition scores were collected using a matrix sentences test and neural …
University of Texas at Dallas（德克萨斯大学达拉斯分校）；Southern University of Science and Technology（南方科技大学）
Electrical hearing by cochlear implants (CIs) may be fundamentally different from acoustic hearing by normal-hearing (NH) listeners, presumably showing unequal speech quality perception in various noise environments. Noise reduction (NR) algorithms used in CI reduce the noise in favor of signal-to-noise ratio (SNR), regardless of plausible accompanying distortions that may degrade the speech quality perception. To gain better understanding of CI speech quality perception, the present work aimed investigating speech quality perception in a diverse noise conditions, including factors of speech/n…
Acoustics Research Institute, Austrian Academy of Sciences（奥地利科学院声学研究所）
Sensitivity to ITDs is important for sound localization. Normal-hearing listeners benefit from across-frequency processing, as seen with improved ITD thresholds when consistent ITD cues are presented over a range of frequency channels compared to when ITD information is only presented in a single frequency channel. This study aimed to clarify whether cochlear-implant (CI) listeners can make use of similar processing when being stimulated with multiple interaural electrode pairs transmitting consistent ITD information. ITD thresholds for unmodulated, 100-pulse-per-second pulse trains were measu…
From Spikes to Speech: NeuroVoc -- A Biologically Plausible Vocoder Framework for Auditory Perception and Cochlear Implant Simulation
Leiden Institute of Advanced Computer Science（莱顿高级计算机科学研究所）；Department of Otorhinolaryngology（耳鼻喉科部门）；Leiden Institute for Brain and Cognition（莱顿脑认知研究所）；Bioelectronics group, EEMCS（生物电子组，EEMCS）
Many hearing-impaired listeners struggle to localize sounds due to poor availability of binaural cues. Listeners with a cochlear implant and a contralateral hearing aid -- so-called bimodal listeners -- are amongst the worst performers, as both interaural time and level differences are poorly transmitted. We present a new method to enhance head shadow in the low frequencies. Head shadow enhancement is achieved with a fixed beamformer with contralateral attenuation in each ear. The method results in interaural level differences which vary monotonically with angle. It also improves low-frequency…
Federal University of Santa Catarina（圣卡塔琳娜联邦大学）；Wavetech ST（Wavetech ST公司）
This work presents a unified framework for defining a family of noise reduction techniques for speech enhancement applications. The proposed approach provides a unique theoretical foundation for some widely-applied soft and hard time-frequency masks, which encompasses the well-known Wiener filter and the heuristically-designed Binary mask. These techniques can now be considered as optimal solutions of the same minimization problem. The proposed cost function is defined by two design parameters that not only establish a desired trade-off between noise reduction and speech distortion, but also p…
Institute of Information Science, Academia Sinica（中央研究院资讯科学研究所）
The combined electric and acoustic stimulation (EAS) has demonstrated better speech recognition than conventional cochlear implant (CI) and yielded satisfactory performance under quiet conditions. However, when noise signals are involved, both the electric signal and the acoustic signal may be distorted, thereby resulting in poor recognition performance. To suppress noise effects, speech enhancement (SE) is a necessary unit in EAS devices. Recently, a time-domain speech enhancement algorithm based on the fully convolutional neural networks (FCN) with a short-time objective intelligibility (STO…
