In multi speakers environments, cochlear implant (CI) users may attend to a target sound source in a different manner from the normal hearing (NH) individuals during a conversation. This study attempted to investigate the effect of conversational sound levels on the mechanisms adopted by CI and NH listeners in selective auditory attention and how it affects their daily conversation. Nine CI users (five bilateral, three unilateral, and one bimodal) and eight NH listeners participated in this study. The behavioral speech recognition scores were collected using a matrix sentences test and neural tracking to speech envelope was recorded using electroencephalography (EEG). Speech stimuli were presented at three different levels (75, 65, and 55 dB SPL) in the presence of two maskers from three spatially separated speakers. Different combinations of assisted/impaired hearing modes were evaluated for CI users and the outcomes were analyzed in three categories: electric hearing only, acoustic hearing only, and electric+acoustic hearing. Our results showed that increasing the conversational sound level degraded the selective auditory attention in electrical hearing. On the other hand, increasing the sound level improved the selective auditory attention for the acoustic hearing group. In NH listeners, however, increasing the sound level did not cause a significant change in the auditory attention. Our result implies that the effect of the sound level on the selective auditory attention varies depending on hearing modes and the loudness control is necessary for the ease of attending to the conversation by CI users.
DeepSpeech models show Human-like Performance and Processing of Cochlear Implant Inputs
Center for Theoretical Neuroscience（理论神经科学中心）；Zuckerman Mind Brain Behavior Institute（扎克曼脑与行为研究所）；Columbia University（哥伦比亚大学）；Google DeepMind（谷歌DeepMind）
Auditory models are commonly used as feature extractors for automatic speech-recognition systems or as front-ends for robotics, machine-hearing and hearing-aid applications. Although auditory models can capture the biophysical and nonlinear properties of human hearing in great detail, these biophysical models are computationally expensive and cannot be used in real-time applications. We present a hybrid approach where convolutional neural networks are combined with computational neuroscience to yield a real-time end-to-end model for human cochlear mechanics, including level-dependent filter tu…
National University of Singapore（新加坡国立大学）；Agency for Science, Technology and Research（新加坡科技研究局）；University of New South Wales（新南威尔士大学）
Auditory front-end is an integral part of a spiking neural network (SNN) when performing auditory cognitive tasks. It encodes the temporal dynamic stimulus, such as speech and audio, into an efficient, effective and reconstructable spike pattern to facilitate the subsequent processing. However, most of the auditory front-ends in current studies have not made use of recent findings in psychoacoustics and physiology concerning human listening. In this paper, we propose a neural encoding and decoding scheme that is optimized for speech processing. The neural encoding scheme, that we call Biologic…
Cochlear implant users struggle to understand speech in reverberant environments. To restore speech perception, artifacts dominated by reverberant reflections can be removed from the cochlear implant stimulus. Artifacts can be identified and removed by applying a matrix of gain values, a technique referred to as time-frequency masking. Gain values are determined by an oracle algorithm that uses knowledge of the undistorted signal to minimize retention of the signal components dominated by reverberant reflections. In practice, gain values are estimated from the distorted signal, with the oracle…
Enhancing spatial hearing with cochlear implants: exploring the role of AI, multimodal interaction and perceptual training
Acoustics Research Institute, Austrian Academy of Sciences（奥地利科学院声学研究所）；Centre de Recherche en Neurosciences de Lyon Inserm（里昂神经科学研究中心（Inserm））；Department of Neural Dynamics and Magnetoencephalography, Hertie Institute for Clinical Brain Research, University of Tübingen（图宾根大学神经动力学与脑磁图部门）；Centre for Integrative Neuroscience, University of Tübingen（图宾根大学整合神经科学中心）；NEMO Labs Nonprofit Kft., Budapest（布达佩斯NEMO实验室（非营利公司））
Speaker recognition is a biometric modality that uses underlying speech information to determine the identity of the speaker. Speaker Identification (SID) under noisy conditions is one of the challenging topics in the field of speech processing, specifically when it comes to individuals with cochlear implants (CI). This study analyzes and quantifies the ability of CI-users to perform speaker identification based on direct electric auditory stimuli. CI users employ a limited number of frequency bands (8 to 22) and use electrodes to directly stimulate the Basilar Membrane/Cochlear in order to re…
Cochlear implant (CI) users have considerable difficulty in understanding speech in reverberant listening environments. Time-frequency (T-F) masking is a common technique that aims to improve speech intelligibility by multiplying reverberant speech by a matrix of gain values to suppress T-F bins dominated by reverberation. Recently proposed mask estimation algorithms leverage machine learning approaches to distinguish between target speech and reverberant reflections. However, the spectro-temporal structure of speech is highly variable and dependent on the underlying phoneme. One way to potent…
Research Center for Information Technology Innovation, Academia Sinica（中央研究院资讯科技创新研究中心）；National Taiwan University（台湾大学）；Institute of Linguistics, Academia Sinica（中央研究院语言学研究所）
Speech perception is key to verbal communication. For people with hearing loss, the capability to recognize speech is restricted, particularly in a noisy environment or the situations without visual cues, such as lip-reading unavailable via phone call. This study aimed to understand the improvement of vocoded speech intelligibility in cochlear implant (CI) simulation through two potential methods: Speech Enhancement (SE) and Audiovisual Integration. A fully convolutional neural network (FCN) using an intelligibility-oriented objective function was recently proposed and proven to effectively fa…
Attempts to develop speech enhancement algorithms with improved speech intelligibility for cochlear implant (CI) users have met with limited success. To improve speech enhancement methods for CI users, we propose to perform speech enhancement in a cochlear filter-bank feature space, a feature-set specifically designed for CI users based on CI auditory stimuli. We leverage a convolutional neural network (CNN) to extract both stationary and non-stationary components of environmental acoustics and speech. We propose three CNN architectures: (1) vanilla CNN that directly generates the enhanced sig…
École Normale Supérieure, PSL University（巴黎高等师范学院，巴黎文理研究大学）；CNRS（法国国家科学研究中心）；University of Rochester（罗切斯特大学）；Technical University of Denmark（丹麦技术大学）；McMaster University（麦克马斯特大学）；Ghent University（根特大学）；Acoustics Research Institute, Austrian Academy of Sciences（奥地利科学院声学研究所）
A number of auditory models have been developed using diverging approaches, either physiological or perceptual, but they share comparable stages of signal processing, as they are inspired by the same constitutive parts of the auditory system. We compare eight monaural models that are openly accessible in the Auditory Modelling Toolbox. We discuss the considerations required to make the model outputs comparable to each other, as well as the results for the following model processing stages or their equivalents: Outer and middle ear, cochlear filter bank, inner hair cell, auditory nerve synapse,…
University of Texas at Dallas（德克萨斯大学达拉斯分校）；Southern University of Science and Technology（南方科技大学）
Electrical hearing by cochlear implants (CIs) may be fundamentally different from acoustic hearing by normal-hearing (NH) listeners, presumably showing unequal speech quality perception in various noise environments. Noise reduction (NR) algorithms used in CI reduce the noise in favor of signal-to-noise ratio (SNR), regardless of plausible accompanying distortions that may degrade the speech quality perception. To gain better understanding of CI speech quality perception, the present work aimed investigating speech quality perception in a diverse noise conditions, including factors of speech/n…
End-to-end audio-visual learning for cochlear implant sound coding simulations in noisy environments
Graduate Institute of Electronics Engineering（电子工程研究所）；the Department of Electrical Engineering, National Taiwan University, Taipei 106319, Taiwan（国立台湾大学电子工程系）；Research Center for Information Technology Innovation Technology, Academia Sinica, Taipei 115201, Taiwan（中科院资讯科技创新研究中心）；Department of Electrical Engineering, Chung Yuan Christian University, Taoyuan 320314, Taiwan（Chung Yuan Christian University 电子工程系）
Pruning-aware Loss Functions for STOI-Optimized Pruned Recurrent Autoencoders for the Compression of the Stimulation Patterns of Cochlear Implants at Zero Delay
From Spikes to Speech: NeuroVoc -- A Biologically Plausible Vocoder Framework for Auditory Perception and Cochlear Implant Simulation
Leiden Institute of Advanced Computer Science（莱顿高级计算机科学研究所）；Department of Otorhinolaryngology（耳鼻喉科部门）；Leiden Institute for Brain and Cognition（莱顿脑认知研究所）；Bioelectronics group, EEMCS（生物电子组，EEMCS）
ElectrodeNet -- A Deep Learning Based Sound Coding Strategy for Cochlear Implants
National Central University（国立中央大学）；Research Center for Information Technology Innovation, Academia Sinica（中央研究院资讯科技创新研究中心）；National Cheng Kung University（国立成功大学）；Chung Yuan Christian University（中原基督教大学）
Many hearing-impaired listeners struggle to localize sounds due to poor availability of binaural cues. Listeners with a cochlear implant and a contralateral hearing aid -- so-called bimodal listeners -- are amongst the worst performers, as both interaural time and level differences are poorly transmitted. We present a new method to enhance head shadow in the low frequencies. Head shadow enhancement is achieved with a fixed beamformer with contralateral attenuation in each ear. The method results in interaural level differences which vary monotonically with angle. It also improves low-frequency…
Fudan University（复旦大学）；National Institute of Informatics (NII)（国立情报学研究所）；Shanghai Key Laboratory of Intelligent Information Processing, Fudan University（复旦大学上海市智能信息处理重点实验室）
Musical audio is generally composed of three physical properties: frequency, time and magnitude. Interestingly, human auditory periphery also provides neural codes for each of these dimensions to perceive music. Inspired by these intrinsic characteristics, a frequency-temporal attention network is proposed to mimic human auditory for singing melody extraction. In particular, the proposed model contains frequency-temporal attention modules and a selective fusion module corresponding to these three physical properties. The frequency attention module is used to select the same activation frequenc…
