Modeling of Speech-dependent Own Voice Transfer Characteristics for Hearables with In-ear Microphones
Fraunhofer Institute for Digital Media Technology IDMT（弗劳恩霍夫数字媒体技术研究所 IDMT）；Carl von Ossietzky Universität Oldenburg（奥尔登堡大学）
Mattes Ohlenbusch, Christian Rollwage, Simon Doclo
备注：
20 pages, 11 figures; Extended version of arXiv:2309.08294 (more detailed description of the problem, additional models considered, more systematic evaluation conducted on a different, larger dataset) -> Updated version (20th march 2024): major changes after internal review; in submission
Many hearables contain an in-ear microphone, which may be used to capture the own voice of its user. However, due to the hearable occluding the ear canal, the in-ear microphone mostly records body-conducted speech, typically suffering from band-limitation effects and amplification at low frequencies. Since the occlusion effect is determined by the ratio between the air-conducted and body-conducted components of own voice, the own voice transfer characteristics between the outer face of the hearable and the in-ear microphone depend on the speech content and the individual talker. In this paper, we propose a speech-dependent model of the own voice transfer characteristics based on phoneme recognition, assuming a linear time-invariant relative transfer function for each phoneme. We consider both individual models as well as models averaged over several talkers. Experimental results based on recordings with a prototype hearable show that the proposed speech-dependent model enables to simulate in-ear signals more accurately than a speech-independent model in terms of technical measures, especially under utterance mismatch and talker mismatch. Additionally, simulation results show that talker-averaged models generalize better to different talkers than individual models.
It was shown in our earlier work that the measurement error in the microphone position affected the room impulse response (RIR) which in turn affected the single-channel close microphone and multi-channel distant microphone speech recognition. In this paper, as an extension, we systematically study to identify the optimal location of the microphone, given an approximate and hence erroneous location of the microphone in 3D space. The primary idea is to use Monte-Carlo technique to generate a large number of random microphone positions around the erroneous microphone position and select the micr…
We investigate a speech enhancement method based on the binaural coherence-to-diffuse power ratio (CDR), which preserves auditory spatial cues for maskers and a broadside target. Conventional CDR estimators typically rely on a mathematical coherence model of the desired signal and/or diffuse noise field in their formulation, which may influence their accuracy in natural environments. This work proposes a new robust and parameterized directional binaural CDR estimator. The estimator is calculated in the time-frequency domain and is based on a geometrical interpretation of the spatial coherence …
WhisperMask: A Noise Suppressive Mask-Type Microphone for Whisper Speech
The University of Tokyo（东京大学）；National Institute of Advanced Industrial Science and Technology（独立行政法人产业技术综合研究所）；Sony Computer Science Laboratory（索尼计算机科学实验室）
Beijing University of Posts and Telecommunications（北京邮电大学）；University of Washington（华盛顿大学）；Harman International（哈曼国际）
Voice communication using the air conduction microphone in noisy environments suffers from the degradation of speech audibility. Bone conduction microphones (BCM) are robust against ambient noises but suffer from limited effective bandwidth due to their sensing mechanism. Although existing audio super resolution algorithms can recover the high frequency loss to achieve high-fidelity audio, they require considerably more computational resources than available in low-power hearable devices. This paper proposes the first-ever real-time on-chip speech audio super resolution system for BCM. To acco…
Fraunhofer Institute for Digital Media Technology IDMT（弗劳恩霍夫数字媒体技术研究所（IDMT））；University of Oldenburg（奥尔登堡大学）
In-ear microphones in hearing protection devices can be utilized to capture the own voice speech of the person wearing the devices in noisy environments. Since in-ear recordings of the own voice are typically band-limited, an own voice reconstruction system is required to recover clean broadband speech from the in-ear signals. However, the availability of speech data for this scenario is typically limited due to device-specific transfer characteristics and the need to collect data from in-situ measurements. In this paper, we apply a deep learning-based bandwidth-extension system to the own voi…
Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学）；Harokopio University of Athens（雅典哈罗科皮奥大学）
The importance of automated and objective monitoring of dietary behavior is becoming increasingly accepted. The advancements in sensor technology along with recent achievements in machine-learning--based signal-processing algorithms have enabled the development of dietary monitoring solutions that yield highly accurate results. A common bottleneck for developing and training machine learning algorithms is obtaining labeled data for training supervised algorithms, and in particular ground truth annotations. Manual ground truth annotation is laborious, cumbersome, can sometimes introduce errors,…
University of Oldenburg（奥尔登堡大学）；Jade Hochschule（Jade应用科学大学）
In this paper we consider an in-ear headphone equipped with an external microphone and aim to minimize the sound pressure at the ear drum by means of a fixed feedforward ANC controller. Based on measured acoustic paths to predict the sound pressure generated by external sources and the headphone at the ear drum, the FIR filter coefficients of the ANC controller are optimized for different sound fields. Due to the acoustic feedback path between the loudspeaker and the microphone, a stability constraint based on the Nyquist stability criterion is introduced. Performance degradations due to reins…
We present a database of acoustic transfer functions of the Hearpiece, an openly available multi-microphone multi-driver in-the-ear earpiece for hearing device research. The database includes HRTFs for 87 incidence directions as well as responses of the drivers, all measured at the four microphones of the Hearpiece as well as the eardrum in the occluded and open ear. The transfer functions were measured in both ears of 25 human subjects and a KEMAR with anthropometric pinnae for five reinsertions of the device. We describe the measurements of the database and analyse derived acoustic parameter…
Active noise cancellation on open-ear smart glasses
Department of Electrical and Computer Engineering, Carnegie Mellon University（卡内基梅隆大学电气与计算机工程系）；Department of Medical Physics and Acoustics, Carl von Ossietzky Universität Oldenburg（奥尔登堡大学医学物理与声学系）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
In this paper we consider an in-ear headphone equipped with an inner microphone and multiple loudspeakers and we propose an optimization procedure with a convex objective function to derive a fixed multi-loudspeaker ANC controller aiming at minimizing the sound pressure at the ear drum. Based on the virtual microphone arrangement (VMA) technique and measured acoustic paths between the loudspeakers and the ear drum, the FIR filters of the ANC controller are jointly optimized to minimize the power spectral density at the ear drum, subject to design and stability constraints. For an in-ear headph…
Paris Brain Institute - ICM（巴黎大脑研究所 - ICM）；Institut Polytechnique de Paris（巴黎综合理工学院）；Sorbonne University（索邦大学）；Concordia University（康考迪亚大学）
Many articles have used voice analysis to detect Parkinson's disease (PD), but few have focused on the early stages of the disease and the gender effect. In this article, we have adapted the latest speaker recognition system, called x-vectors, in order to detect an early stage of PD from voice analysis. X-vectors are embeddings extracted from a deep neural network, which provide robust speaker representations and improve speaker recognition when large amounts of training data are used. Our goal was to assess whether, in the context of early PD detection, this technique would outperform the mor…
ABSE-NET: A Lightweight Neural Model for Active Binaural Speech Enhancement in Open-Fit Hearing Aids
College of Computer Science, Inner Mongolia University（内蒙古大学计算机学院）；College of Electronic and Information Engineering, Inner Mongolia University（内蒙古大学电子与信息工程学院）
University of Cambridge（剑桥大学）；Samsung AI（三星人工智能）；Avignon University（阿维尼翁大学）
The ubiquity of microphone-enabled devices has lead to large amounts of unlabelled audio data being produced at the edge. The integration of self-supervised learning (SSL) and federated learning (FL) into one coherent system can potentially offer data privacy guarantees while also advancing the quality and robustness of speech representations. In this paper, we provide a first-of-its-kind systematic study of the feasibility and complexities for training speech SSL models under FL scenarios from the perspective of algorithms, hardware, and systems limits. Despite the high potential of their com…
