Approximately 6.2% of the world's population (466 million people) suffer from disabling hearing impairment [1]. Hearing impairment impacts negatively on one's education, financial success [2][3], cognitive development in childhood [4], including increased risk of dementia in older adulthood [5]. Lack of or reduced social interaction due to hearing impairment affects creating or maintaining healthy relationships at home, school and work [5]. Hence, hearing impairment genuinely affects the overall quality of life and wellbeing.
The cocktail party effect, which is a healthy hearing individual's ability to understand one voice in a cacophony of other voices or sounds, is an important ability lacking in people with hearing impairment. This inability results in difficulties with simple daily activities such as partaking in group discussions or conversing in noisy restaurants [6]. This smart hearing aid aims to provide much-needed assistance with understanding speech in noisy environments. For example, if a person wants to partake in a group discussion, he/she needs to place the microphone array based unit on a flat surface in front of him/her, such as a table. When conversations take place, the microphone array will capture and process sound from all directions, intelligently prioritise and provide the lead speaker's voice by suppressing unwanted noises, including speeches of other people. This device selects and alternates voices between speakers automatically using voice algorithms. Additionally, the user has the option of further fine-tuning the acoustic parameters as needed through a smartphone interface. This paper describes the development and functions of this new Smart Hearing Aid.
In this letter, we derive a new super Gaussian Joint Maximum a Posteriori based single microphone speech enhancement gain function. The developed Speech Enhancement method is implemented on a smartphone, and this arrangement functions as an assistive device to hearing aids. We introduce a tradeoff parameter in the derived gain function that allows the smartphone user to customize their listening preference, by controlling the amount of noise suppression and speech distortion in real-time based on their level of hearing comfort perceived in noisy real world acoustic environment. Objective quali…
Fraunhofer Institute for Digital Media Technology IDMT（弗劳恩霍夫数字媒体技术研究所（IDMT））；University of Oldenburg（奥尔登堡大学）
In-ear microphones in hearing protection devices can be utilized to capture the own voice speech of the person wearing the devices in noisy environments. Since in-ear recordings of the own voice are typically band-limited, an own voice reconstruction system is required to recover clean broadband speech from the in-ear signals. However, the availability of speech data for this scenario is typically limited due to device-specific transfer characteristics and the need to collect data from in-situ measurements. In this paper, we apply a deep learning-based bandwidth-extension system to the own voi…
The proposed Circular statistics-based Inter-Microphone Phase difference estimation Localizer (CIMPL) method is tailored toward binaural hearing aid systems with microphone arrays in each unit. The method utilizes the circular statistics (circular mean and circular variance) of inter-microphone phase difference (IPD) across different microphone pairs. These IPDs are firstly mapped to time delays through a variance-weighted linear fit, then mapped to azimuth direction-of-arrival (DoA) and lastly information of different microphone pairs is combined. The variance is carried through the different…
Assisted RTF-Vector-Based Binaural Direction of Arrival Estimation Exploiting a Calibrated External Microphone Array
University of Oldenburg, Department of Medical Physics and Acoustics（奥尔登堡大学医学物理与声学系）；Cluster of Excellence Hearing4all, Oldenburg, Germany（卓越中心Hearing4all）
Many personal devices have transitioned from visual-controlled interfaces to speech-controlled interfaces to reduce device costs and interactive friction. This transition has been hastened by the increasing capabilities of speech-controlled interfaces, e.g., Amazon Echo or Apple's Siri. A consequence is that people who are deaf or hard of hearing (DHH) may be unable to use these speech-controlled devices. We show that deaf speech has a high error rate compared to hearing speech, in commercial speech-controlled interfaces. Deaf speech had approximately a 78% word error rate (WER) compared to a …
The goal of this work is to train discriminative cross-modal embeddings without access to manually annotated data. Recent advances in self-supervised learning have shown that effective representations can be learnt from natural cross-modal synchrony. We build on earlier work to train embeddings that are more discriminative for uni-modal downstream tasks. To this end, we propose a novel training strategy that not only optimises metrics across modalities, but also enforces intra-class feature separation within each of the modalities. The effectiveness of the method is demonstrated on two downstr…
In this paper, we propose NEC (Neural Enhanced Cancellation), a defense mechanism, which prevents unauthorized microphones from capturing a target speaker's voice. Compared with the existing scrambling-based audio cancellation approaches, NEC can selectively remove a target speaker's voice from a mixed speech without causing interference to others. Specifically, for a target speaker, we design a Deep Neural Network (DNN) model to extract high-level speaker-specific but utterance-independent vocal features from his/her reference audios. When the microphone is recording, the DNN generates a shad…
This paper introduces the Voices Obscured In Complex Environmental Settings (VOICES) corpus, a freely available dataset under Creative Commons BY 4.0. This dataset will promote speech and signal processing research of speech recorded by far-field microphones in noisy room conditions. Publicly available speech corpora are mostly composed of isolated speech at close-range microphony. A typical approach to better represent realistic scenarios, is to convolve clean speech with noise and simulated room response for model training. Despite these efforts, model performance degrades when tested agains…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Augmented listening devices such as hearing aids often perform poorly in noisy and reverberant environments with many competing sound sources. Large distributed microphone arrays can improve performance, but data from remote microphones often cannot be used for delay-constrained real-time processing. We present a cooperative audio source separation and enhancement system that leverages wearable listening devices and other microphone arrays spread around a room. The full distributed array is used to separate sound sources and estimate their statistics. Each listening device uses these statistic…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Augmented listening devices, such as hearing aids and augmented reality headsets, enhance human perception by changing the sounds that we hear. Microphone arrays can improve the performance of listening systems in noisy environments, but most array-based listening systems are designed to isolate a single sound source from a mixture. This work considers a source-remixing filter that alters the relative level of each source independently. Remixing rather than separating sounds can help to improve perceptual transparency: it causes less distortion to the signal spectrum and especially to the inte…
WhisperMask: A Noise Suppressive Mask-Type Microphone for Whisper Speech
The University of Tokyo（东京大学）；National Institute of Advanced Industrial Science and Technology（独立行政法人产业技术综合研究所）；Sony Computer Science Laboratory（索尼计算机科学实验室）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
In real-time listening enhancement applications, such as hearing aid signal processing, sounds must be processed with no more than a few milliseconds of delay to sound natural to the listener. Listening devices can achieve better performance with lower delay by using microphone arrays to filter acoustic signals in both space and time. Here, we analyze the tradeoff between delay and squared-error performance of causal multichannel Wiener filters for microphone array noise reduction. We compute exact expressions for the delay-error curves in two special cases and present experimental results fro…
Invariance to microphone array configuration is a rare attribute in neural beamformers. Filter-and-sum (FS) methods in this class define the target signal with respect to a reference channel. However, this not only complicates formulation in reverberant conditions but also the network, which must have a mechanism to infer what the reference channel is. To address these issues, this study presents Delay Filter-and-Sum Network (DFSNet), a steerable neural beamformer invariant to microphone number and array geometry for causal speech enhancement. In DFSNet, acquired signals are first steered towa…
Universität Oldenburg（奥尔登堡大学）；HörTech gGmbH（HörTech有限责任公司）
Loudspeaker-based spatial audio reproduction schemes are increasingly used for evaluating hearing aids in complex acoustic conditions. To further establish the feasibility of this approach, this study investigated the interaction between spatial resolution of different reproduction methods and technical and perceptual hearing aid performance measures using computer simulations. Three spatial audio reproduction methods -- discrete speakers, vector base amplitude panning and higher order ambisonics -- were compared in regular circular loudspeaker arrays with 4 to 72 channels. The influence of re…
Nowadays, speech is becoming a more common, if not standard, interface to technology. This can be seen in the trend of technology changes over the years. Increasingly, voice is used to control programs, appliances and personal devices within homes, cars, workplaces, and public spaces through smartphones and home assistant devices using Amazon's Alexa, Google's Assistant and Apple's Siri, and other proliferating technologies. However, most speech interfaces are not accessible for Deaf and Hard-of-Hearing (DHH) people. In this paper, performances of current Automatic Speech Recognition (ASR) wit…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
We present an open-access dataset of over 8000 acoustic impulse from 160 microphones spread across the body and affixed to wearable accessories. The data can be used to evaluate audio capture and array processing systems using wearable devices such as hearing aids, headphones, eyeglasses, jewelry, and clothing. We analyze the acoustic transfer functions of different parts of the body, measure the effects of clothing worn over microphones, compare measurements from a live human subject to those from a mannequin, and simulate the noise-reduction performance of several beamformers. The results su…
Audatic（Audatic公司）；Charité – Universitätsmedizin Berlin（柏林夏里特医学院）；Technische Universität Berlin（柏林工业大学）；Bernstein Center for Computational Neuroscience Berlin（柏林伯恩斯坦计算神经科学中心）
Almost half a billion people world-wide suffer from disabling hearing loss. While hearing aids can partially compensate for this, a large proportion of users struggle to understand speech in situations with background noise. Here, we present a deep learning-based algorithm that selectively suppresses noise while maintaining speech signals. The algorithm restores speech intelligibility for hearing aid users to the level of control subjects with normal hearing. It consists of a deep network that is trained on a large custom database of noisy speech signals and is further optimized by a neural ar…
Conventional approaches to sound source localization require at least two microphones. It is known, however, that people with unilateral hearing loss can also localize sounds. Monaural localization is possible thanks to the scattering by the head, though it hinges on learning the spectra of the various sources. We take inspiration from this human ability to propose algorithms for accurate sound source localization using a single microphone embedded in an arbitrary scattering structure. The structure modifies the frequency response of the microphone in a direction-dependent way giving each dire…
EEG-informed attended speaker extraction from recorded speech mixtures with application in neuro-steered hearing prostheses
ESAT Laboratory of KU Leuven（KU莱顿大学ESAT实验室）；KU Leuven（KU莱顿大学）；Department of Electrical Engineering (ESAT)（电气工程系（ESAT））；Department of Neurosciences（神经科学系）
The cocktail party problem comprises the challenging task of understanding a speech signal in a complex acoustic environment, where multiple speakers and background noise signals simultaneously interfere with the speech signal of interest. A signal processing algorithm that can effectively increase the speech intelligibility and quality of speech signals in such complicated acoustic situations is highly desirable. Especially for applications involving mobile communication devices and hearing assistive devices. Due to the re-emergence of machine learning techniques, today, known as deep learnin…
Key Laboratory of Modern Acoustics, Institute of Advanced Artificial Intelligence in Nanjing（南京近代声学重点实验室，南京先进人工智能研究院）
A robust method for linear array is proposed to address the difficulty of direction-of-arrival (DOA) estimation in reverberant and noisy environments. A direct-path dominance test based on the onset detection is utilized to extract time-frequency bins containing the direct propagation of the speech. The influence of the transient noise, which severely contaminates the onset test, is mitigated by a proper transient noise determination scheme. Then for voice features, a two-stage procedure is designed based on the extracted bins and an effective dereverberation method, with robust but possibly b…
Algorithms for acoustic source localization and tracking provide estimates of the positional information about active sound sources in acoustic environments and are essential for a wide range of applications such as personal assistants, smart homes, tele-conferencing systems, hearing aids, or autonomous systems. The aim of the IEEE-AASP Challenge on sound source localization and tracking (LOCATA) was to objectively benchmark state-of-the-art localization and tracking algorithms using an open-access data corpus of recordings for scenarios typically encountered in audio and acoustic signal proce…
The hearing sense on a mobile robot is important because it is omnidirectional and it does not require direct line-of-sight with the sound source. Such capabilities can nicely complement vision to help localize a person or an interesting event in the environment. To do so the robot auditory system must be able to work in noisy, unknown and diverse environmental conditions. In this paper we present a robust sound source localization method in three-dimensional space using an array of 8 microphones. The method is based on time delay of arrival estimation. Results show that a mobile robot can loc…
Many hearing-impaired listeners struggle to localize sounds due to poor availability of binaural cues. Listeners with a cochlear implant and a contralateral hearing aid -- so-called bimodal listeners -- are amongst the worst performers, as both interaural time and level differences are poorly transmitted. We present a new method to enhance head shadow in the low frequencies. Head shadow enhancement is achieved with a fixed beamformer with contralateral attenuation in each ear. The method results in interaural level differences which vary monotonically with angle. It also improves low-frequency…
The development of deep learning technology has greatly promoted the performance improvement of automatic speech recognition (ASR) technology, which has demonstrated an ability comparable to human hearing in many tasks. Voice interfaces are becoming more and more widely used as input for many applications and smart devices. However, existing research has shown that DNN is easily disturbed by slight disturbances and makes false recognition, which is extremely dangerous for intelligent voice applications controlled by voice.
Chittagong University of Engineering & Technology（吉大港工程与技术大学）
This paper describes a robotic arm with 5 degrees-of-freedom (DOF) which is controlled by human voice and has been developed in the Mechatronics Laboratory, CUET. This robotic arm is interfaced with a PC by serial communication (RS-232). Users' voice command is captured by a microphone, and this voice is processed by software which is made by Microsoft visual studio. Then the specific signal (obtained by signal processing) is sent to control unit. The main control unit that is used in the robotic arm is a microcontroller whose model no. is PIC18f452. Then Control unit drives the actuators, (Hi…
We present the signal processing framework and some results for the IEEE AASP challenge on acoustic source localization and tracking (LOCATA). The system is designed for the direction of arrival (DOA) estimation in single-source scenarios. The proposed framework consists of four main building blocks: pre-processing, voice activity detection (VAD), localization, tracking. The signal pre-processing pipeline includes the short-time Fourier transform (STFT) of the multichannel input captured by the array and the cross power spectral density (CPSD) matrices estimation. The VAD is calculated with a …
Acoustic noise has adverse effects on human activities. Aside from hearing impairment and stress-related illnesses, it can also interfere with spoken communication, reduce human performance and affect the quality of life. As urbanization is intensifying, the potential benefits of reducing noise pollution in smart-city environments are extensive. Noise levels can be collected and analyzed using a wireless sensor network which can monitor the noise level by using microphones. However, every wireless system struggles in terms of the battery requirements needed for continuous data collection and m…
Usually, hearing impaired people use hearing aids which are implemented with speech enhancement algorithms. Estimation of speech and estimation of nose are the components in single channel speech enhancement system. The main objective of any speech enhancement algorithm is estimation of noise power spectrum for non stationary environment. VAD (Voice Activity Detector) is used to identify speech pauses and during these pauses only estimation of noise. MMSE (Minimum Mean Square Error) speech enhancement algorithm did not enhance the intelligibility, quality and listener fatigues are the perceptu…
