Speaker recognition is a biometric modality that uses underlying speech information to determine the identity of the speaker. Speaker Identification (SID) under noisy conditions is one of the challenging topics in the field of speech processing, specifically when it comes to individuals with cochlear implants (CI). This study analyzes and quantifies the ability of CI-users to perform speaker identification based on direct electric auditory stimuli. CI users employ a limited number of frequency bands (8 to 22) and use electrodes to directly stimulate the Basilar Membrane/Cochlear in order to recognize the speech signal. The sparsity of electric stimulation within the CI frequency range is a prime reason for loss in human speech recognition, as well as SID performance. Therefore, it is assumed that CI-users might be unable to recognize and distinguish a speaker given dependent information such as formant frequencies, pitch etc. which are lost to un-simulated electrodes. To quantify this assumption, the input speech signal is processed using a CI Advanced Combined Encoder (ACE) signal processing strategy to construct the CI auditory electrodogram. The proposed study uses 50 speakers from each of three different databases for training the system using two different classifiers under quiet, and tested under both quiet and noisy conditions. The objective result shows that, the CI users can effectively identify a limited number of speakers. However, their performance decreases when more speakers are added in the system, as well as when noisy conditions are introduced. This information could therefore be used for improving CI-user signal processing techniques to improve human SID.
Cochlear implant users struggle to understand speech in reverberant environments. To restore speech perception, artifacts dominated by reverberant reflections can be removed from the cochlear implant stimulus. Artifacts can be identified and removed by applying a matrix of gain values, a technique referred to as time-frequency masking. Gain values are determined by an oracle algorithm that uses knowledge of the undistorted signal to minimize retention of the signal components dominated by reverberant reflections. In practice, gain values are estimated from the distorted signal, with the oracle…
Enhancing spatial hearing with cochlear implants: exploring the role of AI, multimodal interaction and perceptual training
Acoustics Research Institute, Austrian Academy of Sciences（奥地利科学院声学研究所）；Centre de Recherche en Neurosciences de Lyon Inserm（里昂神经科学研究中心（Inserm））；Department of Neural Dynamics and Magnetoencephalography, Hertie Institute for Clinical Brain Research, University of Tübingen（图宾根大学神经动力学与脑磁图部门）；Centre for Integrative Neuroscience, University of Tübingen（图宾根大学整合神经科学中心）；NEMO Labs Nonprofit Kft., Budapest（布达佩斯NEMO实验室（非营利公司））
Cochlear implant (CI) users have considerable difficulty in understanding speech in reverberant listening environments. Time-frequency (T-F) masking is a common technique that aims to improve speech intelligibility by multiplying reverberant speech by a matrix of gain values to suppress T-F bins dominated by reverberation. Recently proposed mask estimation algorithms leverage machine learning approaches to distinguish between target speech and reverberant reflections. However, the spectro-temporal structure of speech is highly variable and dependent on the underlying phoneme. One way to potent…
Research Center for Information Technology Innovation, Academia Sinica（中央研究院资讯科技创新研究中心）；National Taiwan University（台湾大学）；Institute of Linguistics, Academia Sinica（中央研究院语言学研究所）
Speech perception is key to verbal communication. For people with hearing loss, the capability to recognize speech is restricted, particularly in a noisy environment or the situations without visual cues, such as lip-reading unavailable via phone call. This study aimed to understand the improvement of vocoded speech intelligibility in cochlear implant (CI) simulation through two potential methods: Speech Enhancement (SE) and Audiovisual Integration. A fully convolutional neural network (FCN) using an intelligibility-oriented objective function was recently proposed and proven to effectively fa…
DeepSpeech models show Human-like Performance and Processing of Cochlear Implant Inputs
Center for Theoretical Neuroscience（理论神经科学中心）；Zuckerman Mind Brain Behavior Institute（扎克曼脑与行为研究所）；Columbia University（哥伦比亚大学）；Google DeepMind（谷歌DeepMind）
University of Texas at Dallas（德克萨斯大学达拉斯分校）；Tongmyong University（东明大学）
In multi speakers environments, cochlear implant (CI) users may attend to a target sound source in a different manner from the normal hearing (NH) individuals during a conversation. This study attempted to investigate the effect of conversational sound levels on the mechanisms adopted by CI and NH listeners in selective auditory attention and how it affects their daily conversation. Nine CI users (five bilateral, three unilateral, and one bimodal) and eight NH listeners participated in this study. The behavioral speech recognition scores were collected using a matrix sentences test and neural …
R.V. College of Engineering（R.V.工程学院）；JNT University（JNT大学）
Cochlear implant devices are known to exist since a long time. The purpose of the present work is to develop a speech algorithm for obtaining robust speech. In this paper, the technique of cochlear implant is first introduced, followed by discussions of some of the existing techniques available for obtaining speech. The next section introduces a new technique for obtaining robust speech. The key feature of this technique lies in the use of the advantages of an integrated approach involving the use of an estimation technique such as a kalman filter with non linear filter bank strategy, using Du…
École Normale Supérieure, PSL University（巴黎高等师范学院，巴黎文理研究大学）；CNRS（法国国家科学研究中心）；University of Rochester（罗切斯特大学）；Technical University of Denmark（丹麦技术大学）；McMaster University（麦克马斯特大学）；Ghent University（根特大学）；Acoustics Research Institute, Austrian Academy of Sciences（奥地利科学院声学研究所）
A number of auditory models have been developed using diverging approaches, either physiological or perceptual, but they share comparable stages of signal processing, as they are inspired by the same constitutive parts of the auditory system. We compare eight monaural models that are openly accessible in the Auditory Modelling Toolbox. We discuss the considerations required to make the model outputs comparable to each other, as well as the results for the following model processing stages or their equivalents: Outer and middle ear, cochlear filter bank, inner hair cell, auditory nerve synapse,…
This dissertation covers a single-processor approach to the speech processing pipeline of bilateral Cochlear Implants (CIs). The use of only a single processor to provide binaural stimulation signals overcomes the synchronization problem, which is an existing challenging problem in the deployment of bilateral CI devices. The developed single-processor speech processing pipeline provides CI users with a sense of directionality. Its non-synchronization feature as well as low computational and memory requirements make it a suitable solution for actual deployment. A speech enhancement framework is…
Attempts to develop speech enhancement algorithms with improved speech intelligibility for cochlear implant (CI) users have met with limited success. To improve speech enhancement methods for CI users, we propose to perform speech enhancement in a cochlear filter-bank feature space, a feature-set specifically designed for CI users based on CI auditory stimuli. We leverage a convolutional neural network (CNN) to extract both stationary and non-stationary components of environmental acoustics and speech. We propose three CNN architectures: (1) vanilla CNN that directly generates the enhanced sig…
Pruning-aware Loss Functions for STOI-Optimized Pruned Recurrent Autoencoders for the Compression of the Stimulation Patterns of Cochlear Implants at Zero Delay
End-to-end audio-visual learning for cochlear implant sound coding simulations in noisy environments
Graduate Institute of Electronics Engineering（电子工程研究所）；the Department of Electrical Engineering, National Taiwan University, Taipei 106319, Taiwan（国立台湾大学电子工程系）；Research Center for Information Technology Innovation Technology, Academia Sinica, Taipei 115201, Taiwan（中科院资讯科技创新研究中心）；Department of Electrical Engineering, Chung Yuan Christian University, Taoyuan 320314, Taiwan（Chung Yuan Christian University 电子工程系）
From Spikes to Speech: NeuroVoc -- A Biologically Plausible Vocoder Framework for Auditory Perception and Cochlear Implant Simulation
Leiden Institute of Advanced Computer Science（莱顿高级计算机科学研究所）；Department of Otorhinolaryngology（耳鼻喉科部门）；Leiden Institute for Brain and Cognition（莱顿脑认知研究所）；Bioelectronics group, EEMCS（生物电子组，EEMCS）
Auditory models are commonly used as feature extractors for automatic speech-recognition systems or as front-ends for robotics, machine-hearing and hearing-aid applications. Although auditory models can capture the biophysical and nonlinear properties of human hearing in great detail, these biophysical models are computationally expensive and cannot be used in real-time applications. We present a hybrid approach where convolutional neural networks are combined with computational neuroscience to yield a real-time end-to-end model for human cochlear mechanics, including level-dependent filter tu…
University of Texas at Dallas（德克萨斯大学达拉斯分校）；Southern University of Science and Technology（南方科技大学）
Electrical hearing by cochlear implants (CIs) may be fundamentally different from acoustic hearing by normal-hearing (NH) listeners, presumably showing unequal speech quality perception in various noise environments. Noise reduction (NR) algorithms used in CI reduce the noise in favor of signal-to-noise ratio (SNR), regardless of plausible accompanying distortions that may degrade the speech quality perception. To gain better understanding of CI speech quality perception, the present work aimed investigating speech quality perception in a diverse noise conditions, including factors of speech/n…
Acoustics Research Institute, Austrian Academy of Sciences（奥地利科学院声学研究所）
Sensitivity to ITDs is important for sound localization. Normal-hearing listeners benefit from across-frequency processing, as seen with improved ITD thresholds when consistent ITD cues are presented over a range of frequency channels compared to when ITD information is only presented in a single frequency channel. This study aimed to clarify whether cochlear-implant (CI) listeners can make use of similar processing when being stimulated with multiple interaural electrode pairs transmitting consistent ITD information. ITD thresholds for unmodulated, 100-pulse-per-second pulse trains were measu…
Many hearing-impaired listeners struggle to localize sounds due to poor availability of binaural cues. Listeners with a cochlear implant and a contralateral hearing aid -- so-called bimodal listeners -- are amongst the worst performers, as both interaural time and level differences are poorly transmitted. We present a new method to enhance head shadow in the low frequencies. Head shadow enhancement is achieved with a fixed beamformer with contralateral attenuation in each ear. The method results in interaural level differences which vary monotonically with angle. It also improves low-frequency…
Representing Speech Through Autoregressive Prediction of Cochlear Tokens
Department of Brain；Cognitive Sciences \& McGovern Institute for Brain Research；Department of Computer Science \& Wu Tsai Neurosciences Institute；Department of Psychology
National University of Singapore（新加坡国立大学）；Agency for Science, Technology and Research（新加坡科技研究局）；University of New South Wales（新南威尔士大学）
Auditory front-end is an integral part of a spiking neural network (SNN) when performing auditory cognitive tasks. It encodes the temporal dynamic stimulus, such as speech and audio, into an efficient, effective and reconstructable spike pattern to facilitate the subsequent processing. However, most of the auditory front-ends in current studies have not made use of recent findings in psychoacoustics and physiology concerning human listening. In this paper, we propose a neural encoding and decoding scheme that is optimized for speech processing. The neural encoding scheme, that we call Biologic…
Speech Separation for Hearing-Impaired Children in the Classroom
Radboud University（拉德堡德大学）；Donders Institute for Brain, Cognition, and Behaviour（大脑、认知与行为研究所）；Leiden University Medical Centre（莱顿大学医学中心）；Mortimer B. Zuckerman Mind, Brain, Behavior Institute（莫蒂默·B·祖克erman心智、大脑与行为研究所）；Columbia University（哥伦比亚大学）；Department of Otorhinolaryngology（耳鼻喉科部门）；Leiden Institute for Brain and Cognition（莱顿大脑与认知研究所）；Delft University of Technology（代尔夫特理工大学）
Institute of Information Science, Academia Sinica（中央研究院资讯科学研究所）
The combined electric and acoustic stimulation (EAS) has demonstrated better speech recognition than conventional cochlear implant (CI) and yielded satisfactory performance under quiet conditions. However, when noise signals are involved, both the electric signal and the acoustic signal may be distorted, thereby resulting in poor recognition performance. To suppress noise effects, speech enhancement (SE) is a necessary unit in EAS devices. Recently, a time-domain speech enhancement algorithm based on the fully convolutional neural networks (FCN) with a short-time objective intelligibility (STO…
