Distant speech processing is a challenging task, especially when dealing with the cocktail party effect. Sound source separation is thus often required as a preprocessing step prior to speech recognition to improve the signal to distortion ratio (SDR). Recently, a combination of beamforming and speech separation networks have been proposed to improve the target source quality in the direction of arrival of interest. However, with this type of approach, the neural network needs to be trained in advance for a specific microphone array geometry, which limits versatility when adding/removing microphones, or changing the shape of the array. The solution presented in this paper is to train a neural network on pairs of microphones with different spacing and acoustic environmental conditions, and then use this network to estimate a time-frequency mask from all the pairs of microphones forming the array with an arbitrary shape. Using this mask, the target and noise covariance matrices can be estimated, and then used to perform generalized eigenvalue (GEV) beamforming. Results show that the proposed approach improves the SDR from 4.78 dB to 7.69 dB on average, for various microphone array geometries that correspond to commercially available hardware.
This paper introduces SMP-PHAT, which performs direction of arrival (DoA) of sound estimation with a microphone array by merging pairs of microphones that are parallel in space. This approach reduces the number of pairwise cross-correlation computations, and brings down the number of flops and memory lookups when searching for DoA. Experiments on low-cost hardware with commonly used microphone arrays show that the proposed method provides the same accuracy as the former SRP-PHAT approach, while reducing the computational load by 39% in some cases.
Jade University of Applied Sciences（雅德应用科学大学）；University of Oldenburg（奥尔登堡大学）
Aiming at estimating the direction of arrival (DOA) of a desired speaker in a multi-talker environment using a microphone array, in this paper we propose a signal-informed method exploiting the availability of an external microphone attached to the desired speaker. The proposed method applies a binary mask to the GCC-PHAT input features of a convolutional neural network, where the binary mask is computed based on the power distribution of the external microphone signal. Experimental results for a reverberant scenario with up to four interfering speakers demonstrate that the signal-informed mas…
An approach to the estimation of the Direction of Arrival (DOA) of wide-band signals with a planar microphone array is presented. Our algorithm estimates an unambiguous DOA using a single planar array in which the microphones are placed fairly close together and the sound source is expected to be in the far field. The algorithm uses the ambiguous DOA estimates obtained from microphone pairs in the array to determine an unambiguous DOA estimate for the array as a whole. The required pair-wise DOAs may be calculated using Time Delay Estimations (TDEs), which may in turn be calculated using cross…
Key Laboratory of Modern Acoustics, Institute of Advanced Artificial Intelligence in Nanjing（南京近代声学重点实验室，南京先进人工智能研究院）
A robust method for linear array is proposed to address the difficulty of direction-of-arrival (DOA) estimation in reverberant and noisy environments. A direct-path dominance test based on the onset detection is utilized to extract time-frequency bins containing the direct propagation of the speech. The influence of the transient noise, which severely contaminates the onset test, is mitigated by a proper transient noise determination scheme. Then for voice features, a two-stage procedure is designed based on the extracted bins and an effective dereverberation method, with robust but possibly b…
The proposed Circular statistics-based Inter-Microphone Phase difference estimation Localizer (CIMPL) method is tailored toward binaural hearing aid systems with microphone arrays in each unit. The method utilizes the circular statistics (circular mean and circular variance) of inter-microphone phase difference (IPD) across different microphone pairs. These IPDs are firstly mapped to time delays through a variance-weighted linear fit, then mapped to azimuth direction-of-arrival (DoA) and lastly information of different microphone pairs is combined. The variance is carried through the different…
Assisted RTF-Vector-Based Binaural Direction of Arrival Estimation Exploiting a Calibrated External Microphone Array
University of Oldenburg, Department of Medical Physics and Acoustics（奥尔登堡大学医学物理与声学系）；Cluster of Excellence Hearing4all, Oldenburg, Germany（卓越中心Hearing4all）
Direction of arrival estimation from distant microphone data using single frequency filtering
International Institute of Information Technology, Hyderabad（印度海得拉巴信息科技学院）；University of Southern California, Los Angeles（美国南加州大学）；Ghent University - imec（根特大学-imec）；Koneru Lakshmaiah Education Foundation（孔鲁拉克希马亚教育基金会）
International Audio Laboratories Erlangen（埃尔朗根国际音频实验室）；Fraunhofer Institute for Integrated Circuits (IIS)（弗劳恩霍夫集成电路研究所（IIS））
In a recent work on direction-of-arrival (DOA) estimation of multiple speakers with convolutional neural networks (CNNs), the phase component of short-time Fourier transform (STFT) coefficients of the microphone signal is given as input and small filters are used to learn the phase relations between neighboring microphones. Due to this chosen filter size, $M-1$ convolution layers are required to achieve the best performance for a microphone array with M microphones. For arrays with large number of microphones, this requirement leads to a high computational cost making the method practically in…
University of California San Diego（加州大学圣迭戈分校）；Université de Sherbrooke（舍布鲁克大学）
We present a system for localizing sound sources in a room with several ad-hoc microphone arrays. Each circular array performs direction of arrival (DOA) estimation independently using commercial software. The DOAs are fed to a fusion center, concatenated, and used to perform the localization based on two proposed methods, which require only few labeled source locations (anchor points) for training. The first proposed method is based on principal component analysis (PCA) of the observed DOA and does not require any knowledge of anchor points. The array cluster can then perform localization on …
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Microphone arrays are usually assumed to have rigid geometries: the microphones may move with respect to the sound field but remain fixed relative to each other. However, many useful arrays, such as those in wearable devices, have sensors that can move relative to each other. We compare two approaches to beamforming with deformable microphone arrays: first, by explicitly tracking the geometry of the array as it changes over time, and second, by designing a time-invariant beamformer based on the second-order statistics of the moving array. The time-invariant approach is shown to be appropriate …
Unlike model-based direction of arrival (DoA) estimation algorithms, supervised learning-based DoA estimation algorithms based on deep neural networks (DNNs) are usually trained for one specific microphone array geometry, resulting in poor performance when applied to a different array geometry. In this paper we illustrate the fundamental difference between supervised learning-based and model-based algorithms leading to this sensitivity. Aiming at designing a supervised learning-based DoA estimation algorithm that generalizes well to different array geometries, in this paper we propose a geomet…
In this paper we consider a binaural hearing aid setup, where in addition to the head-mounted microphones an external microphone is available. For this setup, we investigate the performance of several relative transfer function (RTF) vector estimation methods to estimate the direction of arrival (DOA) of the target speaker in a noisy and reverberant acoustic environment. More in particular, we consider the state-of-the-art covariance whitening (CW) and covariance subtraction (CS) methods, either incorporating the external microphone or not, and the recently proposed spatial coherence (SC) meth…
Measured acoustic data can be contaminated by noise. This typically happens when microphones are mounted in a wind tunnel wall or on the fuselage of an aircraft, where hydrodynamic pressure fluctuations of the Turbulent Boundary Layer (TBL) can mask the acoustic pressures of interest. For measurements done with an array of microphones, methods exist for denoising the acoustic data. Use is made of the fact that the noise is usually concentrated in the diagonal of the Cross-Spectral Matrix, because of the short spatial coherence of TBL noise. This paper reviews several existing denoising methods…
Since space-domain information can be utilized, microphone array beamforming is often used to enhance the quality of the speech by suppressing directional disturbance. However, with the increasing number of microphone, the complexity would be increased. In this paper, a concise beamforming scheme using Maximum Signal-to-Noise Ratio (SNR) filter is proposed to reduce the beamforming complexity. The maximum SNR filter is implemented by using the estimated direction-of-arrival (DOA) of the speech source localization (SSL) and the solving method of independent vector analysis (IVA). Our experiment…
Microsoft Corporation（微软公司）；University of Oldenburg（奥尔登堡大学）
Many deep learning techniques are available to perform source separation and reduce background noise. However, designing an end-to-end multi-channel source separation method using deep learning and conventional acoustic signal processing techniques still remains challenging. In this paper we propose a direction-of-arrival-driven beamforming network (DBnet) consisting of direction-of-arrival (DOA) estimation and beamforming layers for end-to-end source separation. We propose to train DBnet using loss functions that are solely based on the distances between the separated speech signals and the t…
The binaural minimum-variance distortionless-response (BMVDR) beamformer is a well-known noise reduction algorithm that can be steered using the relative transfer function (RTF) vector of the desired speech source. Exploiting the availability of an external microphone that is spatially separated from the head-mounted microphones, an efficient method has been recently proposed to estimate the RTF vector in a diffuse noise field. When multiple external microphones are available, different RTF vector estimates can be obtained by using this method for each external microphone. In this paper, we pr…
A two-stage multi-channel speech enhancement method is proposed which consists of a novel adaptive beamformer, Hybrid Minimum Variance Distortionless Response (MVDR), Isotropic-MVDR (Iso), and a novel multi-channel spectral Principal Components Analysis (PCA) denoising. In the first stage, the Hybrid-MVDR performs multiple MVDRs using a dictionary of pre-defined noise field models and picks the minimum-power outcome, which benefits from the robustness of signal-independent beamforming and the performance of adaptive beamforming. In the second stage, the outcomes of Hybrid and Iso are jointly u…
Massachusetts Institute of Technology（麻省理工学院）；University of Surrey（萨里大学）
This paper proposes sound event localization and detection methods from multichannel recording. The proposed system is based on two Convolutional Recurrent Neural Networks (CRNNs) to perform sound event detection (SED) and time difference of arrival (TDOA) estimation on each pair of microphones in a microphone array. In this paper, the system is evaluated with a four-microphone array, and thus combines the results from six pairs of microphones to provide a final classification and a 3-D direction of arrival (DOA) estimate. Results demonstrate that the proposed approach outperforms the DCASE 20…
Tianjin University（天津大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Beijing Xiaoju Technology Co., Ltd.（北京小桔科技有限公司）
Recently, many deep learning based beamformers have been proposed for multi-channel speech separation. Nevertheless, most of them rely on extra cues known in advance, such as speaker feature, face image or directional information. In this paper, we propose an end-to-end beamforming network for direction guided speech separation given merely the mixture signal, namely MIMO-DBnet. Specifically, we design a multi-channel input and multiple outputs architecture to predict the direction-of-arrival based embeddings and beamforming weights for each source. The precisely estimated directional embeddin…
International Audio Laboratories Erlangen（埃尔朗根国际音频实验室）；Friedrich-Alexander-University Erlangen-Nuremberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Fraunhofer IIS（弗劳恩霍夫集成电路研究所）
The direction-of-arrival (DOA) of sound sources is an essential acoustic parameter used, e.g., for multi-channel speech enhancement or source tracking. Complex acoustic scenarios consisting of sources-of-interest, interfering sources, reverberation, and noise make the estimation of the DOAs corresponding to the sources-of-interest a challenging task. Recently proposed attention mechanisms allow DOA estimators to focus on the sources-of-interest and disregard interference and noise, i.e., they are signal-aware. The attention is typically obtained by a deep neural network (DNN) from a short-time…
The present work deals with a new passive system for real-time detection, classification and direction of arrival estimator of Unmanned Aerial Vehicles (UAVs). The proposed system composed of a very low cost hardware components, comprises two different arrays of three or six-microphones, non-linear amplification and filtering of the analog acoustic signal, avoiding also the saturation effect in case where the UAV is located nearby to the microphones. Advance array processing methods are used to detect and locate the wide-band sources in the near and far-field including array calibration and en…
Acoustic signal acts as an essential input to many systems. However, the pure acoustic signal is very difficult to extract, especially in noisy environments. Existing beamforming systems are able to extract the signal transmitted from certain directions. However, since microphones are centrally deployed, these systems have limited coverage and low spatial resolution. We overcome the above limitations and present ChordMics, a distributed beamforming system. By leveraging the spatial diversity of the distributed microphones, ChordMics is able to extract the acoustic signal from arbitrary points.…
International Audio Laboratories Erlangen（埃尔朗根国际音频实验室）；University of Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）；Fraunhofer IIS（弗劳恩霍夫集成电路研究所）
Supervised learning based methods for source localization, being data driven, can be adapted to different acoustic conditions via training and have been shown to be robust to adverse acoustic environments. In this paper, a convolutional neural network (CNN) based supervised learning method for estimating the direction-of-arrival (DOA) of multiple speakers is proposed. Multi-speaker DOA estimation is formulated as a multi-class multi-label classification problem, where the assignment of each DOA label to the input feature is treated as a separate binary classification problem. The phase compone…
Spatial Analysis and Synthesis Methods: Subjective and Objective Evaluations Using Various Microphone Arrays in the Auralization of a Critical Listening Room
University of Huddersfield（哈德斯菲尔德大学）；Genelec OY（吉力科技公司）
Research & Development Institute of Northwestern Polytechnical University in Shenzhen（西北工业大学深圳研究院）；Northwestern Polytechnical University（西北工业大学）
Far-field speech processing is an important and challenging problem. In this paper, we propose \textit{deep ad-hoc beamforming}, a deep-learning-based multichannel speech enhancement framework based on ad-hoc microphone arrays, to address the problem. It contains three novel components. First, it combines \textit{ad-hoc microphone arrays} with deep-learning-based multichannel speech enhancement, which reduces the probability of the occurrence of far-field acoustic environments significantly. Second, it groups the microphones around the speech source to a local microphone array by a supervised …
Deep Learning Based Stage-wise Two-dimensional Speaker Localization with Large Ad-hoc Microphone Arrays
School of Marine Science and Technology, Northwestern Polytechnical University（西北工业大学海洋科学与技术学院）；Institute of Artificial Intelligence (TeleAI), China Telecom Corp Ltd（中国电信股份有限公司人工智能研究院（TeleAI））
In this work, we describe limitations of the free-field propagation model for designing broadband beamformers for microphone arrays on a rigid surface. Towards this goal, we describe a general framework for quantifying the microphone array performance in a general wave-field by directly solving the acoustic wave equation. The model utilizes Finite-Element-Method (FEM) for evaluating the response of the microphone array surface to background 3D planar and spherical waves. The effectiveness of the framework is established by designing and evaluating a representative broadband beamformer under re…
This paper introduces a new method referred to as KISS-GEV (for Keep It Super Simple Generalized eigenvalue) beamforming. While GEV beamforming usually relies on deep neural network for estimating target and noise time-frequency masks, this method uses a signal processing approach based on the direction of arrival (DoA) of the target. This considerably reduces the amount of computations involved at test time, and works for speech enhancement in unseen conditions as there is no need to train a neural network with noisy speech. The proposed method can also be used to separate speech from a mixtu…
Directions of arrival (DOA) estimation or localization of sources is an important problem in many applications for which numerous algorithms have been proposed. Most localization methods use block-level processing that combines multiple data snapshots to estimate DOA within a block. The DOAs are assumed to be constant within the block duration. However, these assumptions are often violated due to source motion. In this paper, we propose a signal model that captures the linear variations in DOA within a block. We applied conventional beamforming (CBF) algorithm to this model to estimate linear …
We propose a spatial loss for unsupervised multi-channel source separation. The proposed loss exploits the duality of direction of arrival (DOA) and beamforming: the steering and beamforming vectors should be aligned for the target source, but orthogonal for interfering ones. The spatial loss encourages consistency between the mixing and demixing systems from a classic DOA estimator and a neural separator, respectively. With the proposed loss, we train the neural separators based on minimum variance distortionless response (MVDR) beamforming and independent vector analysis (IVA). We also inves…
We propose a novel information-theoretic approach for evaluating microphone arrays that relies on the array physics and geometry rather than the underlying beamforming algorithm. The analogy between Multiple-Input-Multiple-Output (MIMO) wireless communication channel and the acoustic channel of microphone arrays is exploited to define information measures of microphone arrays, which provide upper bounds of the information rate of the microphone array system.
