To phased microphone array for sound source localization, algorithm with both high computational efficiency and high precision is a persistent pursuit. In this paper convolutional neural network (CNN) a kind of deep learning is preliminarily applied as a new algorithm. At high frequency CNN can reconstruct the sound localizations with excellent spatial resolution as good as DAMAS, within a very short time as short as conventional beamforming. This exciting result means that CNN perfectly finds source distribution directly from cross-spectral matrix without given propagation function in advance, and thus CNN deserves to be further explored as a new algorithm.
Massachusetts Institute of Technology（麻省理工学院）；University of Surrey（萨里大学）
This paper proposes sound event localization and detection methods from multichannel recording. The proposed system is based on two Convolutional Recurrent Neural Networks (CRNNs) to perform sound event detection (SED) and time difference of arrival (TDOA) estimation on each pair of microphones in a microphone array. In this paper, the system is evaluated with a four-microphone array, and thus combines the results from six pairs of microphones to provide a final classification and a 3-D direction of arrival (DOA) estimate. Results demonstrate that the proposed approach outperforms the DCASE 20…
Université de Sherbrooke（舍布鲁克大学）；Interdisciplinary Institute for Technological Innovation (3IT)（跨学科技术创新研究所（3IT））
Human-robot interaction in natural settings requires filtering out the different sources of sounds from the environment. Such ability usually involves the use of microphone arrays to localize, track and separate sound sources online. Multi-microphone signal processing techniques can improve robustness to noise but the processing cost increases with the number of microphones used, limiting response time and widespread use on different types of mobile robots. Since sound source localization methods are the most expensive in terms of computing resources as they involve scanning a large 3D space, …
Conventional sound source localization methods are mostly based on a single microphone array that consists of multiple microphones. They are usually formulated as the estimation of the direction of arrival problem. In this paper, we propose a deep-learning-based end-to-end sound source localization method with ad-hoc microphone arrays, where an ad-hoc microphone array is a set of randomly distributed microphone arrays that collaborate with each other. It can produce two-dimensional locations of speakers with only a single microphone per node. Specifically, we divide a targeted indoor space int…
Spatial Analysis and Synthesis Methods: Subjective and Objective Evaluations Using Various Microphone Arrays in the Auralization of a Critical Listening Room
University of Huddersfield（哈德斯菲尔德大学）；Genelec OY（吉力科技公司）
An approach to the estimation of the Direction of Arrival (DOA) of wide-band signals with a planar microphone array is presented. Our algorithm estimates an unambiguous DOA using a single planar array in which the microphones are placed fairly close together and the sound source is expected to be in the far field. The algorithm uses the ambiguous DOA estimates obtained from microphone pairs in the array to determine an unambiguous DOA estimate for the array as a whole. The required pair-wise DOAs may be calculated using Time Delay Estimations (TDEs), which may in turn be calculated using cross…
Abstract While vision-based localization techniques have been widely studied for small autonomous unmanned vehicles (SAUVs), sound-source localization capabilities have not been fully enabled for SAUVs. This paper presents two novel approaches for SAUVs to perform three-dimensional (3D) multi-sound-sources localization (MSSL) using only the inter-channel time difference (ICTD) signal generated by a self-rotating bi-microphone array. The proposed two approaches are based on two machine learning techniques viz., Density-Based Spatial Clustering of Applications with Noise (DBSCAN) and Random Samp…
In this paper, we propose an effective and robust method of spatial feature extraction for acoustic scene analysis utilizing partially synchronized and/or closely located distributed microphones. In the proposed method, a new cepstrum feature utilizing a graph-based basis transformation to extract spatial information from distributed microphones, while taking into account whether any pairs of microphones are synchronized and/or closely located, is introduced. Specifically, in the proposed graph-based cepstrum, the log-amplitude of a multichannel observation is converted to a feature vector uti…
IPDnet: A Universal Direct-Path IPD Estimation Network for Sound Source Localization
Zhejiang University（浙江大学）；Westlake University（西湖大学）；School of Engineering, Westlake University（西湖大学工程学院）；Institute of Advanced Technology, Westlake Institute for Advanced Study（西湖高等研究院先进技术研究所）
Conventional approaches to sound source localization require at least two microphones. It is known, however, that people with unilateral hearing loss can also localize sounds. Monaural localization is possible thanks to the scattering by the head, though it hinges on learning the spectra of the various sources. We take inspiration from this human ability to propose algorithms for accurate sound source localization using a single microphone embedded in an arbitrary scattering structure. The structure modifies the frequency response of the microphone in a direction-dependent way giving each dire…
This work presents a novel technique that performs both orientation and distance localization of a sound source in a three-dimensional (3D) space using only the interaural time difference (ITD) cue, generated by a newly-developed self-rotational bi-microphone robotic platform. The system dynamics is established in the spherical coordinate frame using a state-space model. The observability analysis of the state-space model shows that the system is unobservable when the sound source is placed with elevation angles of $90$ and $0$ degree. The proposed method utilizes the difference between the az…
Sound Source Distance Estimation in Diverse and Dynamic Acoustic Conditions
Courant Institute of Mathematical Sciences, New York University（纽约大学柯朗数学科学研究所）；Music and Audio Research Lab, New York University（纽约大学音乐与音频研究实验室）；Integrated Design and Media, New York University（纽约大学集成设计与媒体学院）
Nanyang Technological University（南洋理工大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Sound event localization and detection (SELD) consists of two subtasks, which are sound event detection and direction-of-arrival estimation. While sound event detection mainly relies on time-frequency patterns to distinguish different sound classes, direction-of-arrival estimation uses amplitude and/or phase differences between microphones to estimate source directions. As a result, it is often difficult to jointly optimize these two subtasks. We propose a novel feature called Spatial cue-Augmented Log-SpectrogrAm (SALSA) with exact time-frequency mapping between the signal power and the sourc…
This paper introduces SMP-PHAT, which performs direction of arrival (DoA) of sound estimation with a microphone array by merging pairs of microphones that are parallel in space. This approach reduces the number of pairwise cross-correlation computations, and brings down the number of flops and memory lookups when searching for DoA. Experiments on low-cost hardware with commonly used microphone arrays show that the proposed method provides the same accuracy as the former SRP-PHAT approach, while reducing the computational load by 39% in some cases.
The hearing sense on a mobile robot is important because it is omnidirectional and it does not require direct line-of-sight with the sound source. Such capabilities can nicely complement vision to help localize a person or an interesting event in the environment. To do so the robot auditory system must be able to work in noisy, unknown and diverse environmental conditions. In this paper we present a robust sound source localization method in three-dimensional space using an array of 8 microphones. The method is based on time delay of arrival estimation. Results show that a mobile robot can loc…
Universidade Federal do Rio de Janeiro（里约热内卢联邦大学）；Fluminense Federal University（弗鲁米嫩塞联邦大学）；Inha University（仁荷大学）；Hewlett-Packard Laboratories（惠普实验室）
This paper proposes an efficient method based on the steered-response power (SRP) technique for sound source localization using microphone arrays: the volumetric SRP (V-SRP). As compared to the SRP, by deploying a sparser volumetric grid, the V-SRP achieves a significant reduction of the computational complexity without sacrificing the accuracy of the location estimates. By appending a fine search step to the V-SRP, its refined version (RV-SRP) improves on the compromise between complexity and accuracy. Experiments conducted in both simulated- and real-data scenarios demonstrate the benefits o…
Tsinghua University（清华大学）；Michigan State University（密歇根州立大学）
Sound recognition is an important and popular function of smart devices. The location of sound is basic information associated with the acoustic source. Apart from sound recognition, whether the acoustic sources can be localized largely affects the capability and quality of the smart device's interactive functions. In this work, we study the problem of concurrently localizing multiple acoustic sources with a smart device (e.g., a smart speaker like Amazon Alexa). The existing approaches either can only localize a single source, or require deploying a distributed network of microphone arrays to…
ForestIR: Physics-Informed Forest Sound Simulation for Array-Based Bioacoustic Remote Sensing
Department of Statistical Science, Duke University（统计科学系，杜克大学）；Department of Biological and Environmental Science, University of Jyväskylä（生物与环境科学系，于韦斯屈耶大学）；Faculty of Biological and Environmental Sciences, University of Helsinki（生物与环境科学学院，赫尔辛基大学）；Organismal and Evolutionary Biology Research Programme, Faculty of Biological and Environmental Sciences, University of Helsinki（生物与进化生物学研究计划，生物与环境科学学院，赫尔辛基大学）；Digital Services, University of Jyväskylä（数字服务，于韦斯屈耶大学）；Open Science Centre, University of Jyväskylä（开放科学中心，于韦斯屈耶大学）
Deep neural networks have recently led to promising results for the task of multiple sound source localization. Yet, they require a lot of training data to cover a variety of acoustic conditions and microphone array layouts. One can leverage acoustic simulators to inexpensively generate labeled training data. However, models trained on synthetic data tend to perform poorly with real-world recordings due to the domain mismatch. Moreover, learning for different microphone array layouts makes the task more complicated due to the infinite number of possible layouts. We propose to use adversarial l…
Institute of Engineering, Pulchowk Campus（普尔乔克校区工程学院）；Sanothimi Campus, Tribhuvan University（特里布万大学萨诺蒂米校区）
Sound Source Localization (SSL) are used to estimate the position of sound sources. Various methods have been used for detecting sound and its localization. This paper presents a system for stationary sound source localization by cubical microphone array consisting of eight microphones placed on four vertical adjacent faces which is mounted on three wheel omni-directional drive for the inspection and monitoring of the disaster victims in disaster areas. The proposed method localizes sound source on a 3D space by grid search method using Generalized Cross Correlation Phase Transform (GCC-PHAT) …
Algorithms for acoustic source localization and tracking provide estimates of the positional information about active sound sources in acoustic environments and are essential for a wide range of applications such as personal assistants, smart homes, tele-conferencing systems, hearing aids, or autonomous systems. The aim of the IEEE-AASP Challenge on sound source localization and tracking (LOCATA) was to objectively benchmark state-of-the-art localization and tracking algorithms using an open-access data corpus of recordings for scenarios typically encountered in audio and acoustic signal proce…
Peking University（北京大学）；Westlake University（西湖大学）；Westlake Institute for Advanced Study（西湖高等研究院）
Multiple moving sound source localization in real-world scenarios remains a challenging issue due to interaction between sources, time-varying trajectories, distorted spatial cues, etc. In this work, we propose to use deep learning techniques to learn competing and time-varying direct-path phase differences for localizing multiple moving sound sources. A causal convolutional recurrent neural network is designed to extract the direct-path phase difference sequence from signals of each microphone pair. To avoid the assignment ambiguity and the problem of uncertain output-dimension encountered wh…
We present a novel, reflection-aware method for 3D sound localization in indoor environments. Unlike prior approaches, which are mainly based on continuous sound signals from a stationary source, our formulation is designed to localize the position instantaneously from signals within a single frame. We consider direct sound and indirect sound signals that reach the microphones after reflecting off surfaces such as ceilings or walls. We then generate and trace direct and reflected acoustic paths using inverse acoustic ray tracing and utilize these paths with Monte Carlo localization to estimate…
Sounds recorded with smartphones or IoT devices often have partially unreliable observations caused by clipping, wind noise, and completely missing parts due to microphone failure and packet loss in data transmission over the network. In this paper, we investigate the impact of the partially missing channels on the performance of acoustic scene classification using multichannel audio recordings, especially for a distributed microphone array. Missing observations cause not only losses of time-frequency and spatial information on sound sources but also a mismatch between a trained model and eval…
Korea Advanced Institute of Science and Technology (KAIST)（韩国高等科学技术院（KAIST））
A new database of head-related transfer functions (HRTFs) for accurate sound source localization is presented through precise measurement and post-processing in terms of improved frequency bandwidth and causality of head-related impulse responses (HRIRs) for accurate spectral cue (SC) and interaural time difference (ITD), respectively. The improvement effects of the proposed methods on binaural sound localization cues were investigated. To achieve sufficient frequency bandwidth with a single source, a one-way sealed speaker module was designed to obtain wide band frequency response based on el…
This paper addresses the problem of sound-source localization (SSL) with a robot head, which remains a challenge in real-world environments. In particular we are interested in locating speech sources, as they are of high interest for human-robot interaction. The microphone-pair response corresponding to the direct-path sound propagation is a function of the source direction. In practice, this response is contaminated by noise and reverberations. The direct-path relative transfer function (DP-RTF) is defined as the ratio between the direct-path acoustic transfer function (ATF) of the two microp…
An Efficient GPU-based Implementation for Noise Robust Sound Source Localization
Dept. of Systems and Control Engineering, School of Engineering, Tokyo Institute of Technology（东京技术大学系统与控制工程系）；Honda Research Institute Japan Co., Ltd.（本田研究机构日本株式会社）；NTT Software Innovation Center（NTT软件创新中心）；Dept. of Information and Computer Science, Keio University（Keio大学信息与计算机科学系）
A spatial active noise control (ANC) method based on the interpolation of a sound field from reference microphone signals is proposed. In most current spatial ANC methods, a sufficient number of error microphones are required to reduce noise over the target region because the sound field is estimated from error microphone signals. However, in practical applications, it is preferable that the number of error microphones is as small as possible to keep a space in the target region for ANC users. We propose to interpolate the sound field from reference microphones, which are normally placed outsi…
Inspired by the behavior of humans talking in noisy environments, we propose an embodied embedded cognition approach to improve automatic speech recognition (ASR) systems for robots in challenging environments, such as with ego noise, using binaural sound source localization (SSL). The approach is verified by measuring the impact of SSL with a humanoid robot head on the performance of an ASR system. More specifically, a robot orients itself toward the angle where the signal-to-noise ratio (SNR) of speech is maximized for one microphone before doing an ASR task. First, a spiking neural network …
A microphone array can provide a mobile robot with the capability of localizing, tracking and separating distant sound sources in 2D, i.e., estimating their relative elevation and azimuth. To combine acoustic data with visual information in real world settings, spatial correlation must be established. The approach explored in this paper consists of having two robots, each equipped with a microphone array, localizing themselves in a shared reference map using SLAM. Based on their locations, data from the microphone arrays are used to triangulate in 3D the location of a sound source in relation …
ByteDance（字节跳动）；University of Surrey（萨里大学）；University of Technology Sydney（悉尼科技大学）；Tencent AI Lab（腾讯人工智能实验室）
Sound field decomposition predicts waveforms in arbitrary directions using signals from a limited number of microphones as inputs. Sound field decomposition is fundamental to downstream tasks, including source localization, source separation, and spatial audio reproduction. Conventional sound field decomposition methods such as Ambisonics have limited spatial decomposition resolution. This paper proposes a learning-based Neural Sound field Decomposition (NeSD) framework to allow sound field decomposition with fine spatial direction resolution, using recordings from microphone capsules of a few…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Augmented listening devices, such as hearing aids and augmented reality headsets, enhance human perception by changing the sounds that we hear. Microphone arrays can improve the performance of listening systems in noisy environments, but most array-based listening systems are designed to isolate a single sound source from a mixture. This work considers a source-remixing filter that alters the relative level of each source independently. Remixing rather than separating sounds can help to improve perceptual transparency: it causes less distortion to the signal spectrum and especially to the inte…
Sound event localization and detection (SELD) is a combined task of identifying the sound event and its direction. Deep neural networks (DNNs) are utilized to associate them with the sound signals observed by a microphone array. Although ambisonic microphones are popular in the literature of SELD, they might limit the range of applications due to their predetermined geometry. Some applications (including those for pedestrians that perform SELD while walking) require a wearable microphone array whose geometry can be designed to suit the task. In this paper, for the development of such a wearabl…
Institute of Acoustics, Chinese Academy of Sciences（中国科学院声学研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Conventional approaches to sound localization and separation are based on microphone arrays in artificial systems. Inspired by the selective perception of human auditory system, we design a multi-source listening system which can separate simultaneous overlapping sounds and localize the sound sources in three-dimensional space, using only a single microphone with a metamaterial enclosure. The enclosure modifies the frequency response of the microphone in a direction-dependent way by giving each direction a signature. Thus, the information about the location and audio content of sound sources c…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Augmented listening devices such as hearing aids often perform poorly in noisy and reverberant environments with many competing sound sources. Large distributed microphone arrays can improve performance, but data from remote microphones often cannot be used for delay-constrained real-time processing. We present a cooperative audio source separation and enhancement system that leverages wearable listening devices and other microphone arrays spread around a room. The full distributed array is used to separate sound sources and estimate their statistics. Each listening device uses these statistic…
The equatorial microphone array presented in (Ahrens et al., 2021) computes a spherical harmonic (SH) representation of a sound field based on pressure sensors along the equator of a rigid spherical baffle. The original formulation uses complex-valued SH basis functions. This is inconvenient if the SH representation of the captured sound field is intended to be stored in time domain by means of real-valued audio signals as it is common in the spatial audio format of ambisonics. The present document summarizes the modifications that need to be applied to the mathematical formulation from (Ahren…
Microphone array techniques can improve the acoustic sensing performance on drones, compared to the use of a single microphone. However, multichannel sound acquisition systems are not available in current commercial drone platforms. To encourage the research in drone audition, we present an embedded sound acquisition and recording system with eight microphones and a multichannel sound recorder mounted on a quadcopter. In addition to recording and storing locally the sound from multiple microphones simultaneously, the embedded system can connect wirelessly to a remote terminal to transfer audio…
A data-driven two-microphone method for in-situ sound absorption measurements
Technical University of Munich（慕尼黑工业大学）；KTH Royal Institute of Technology（瑞典皇家理工学院）；Federal University of Santa Maria (UFSM)（圣玛丽亚联邦大学）；Technical University of Denmark（丹麦技术大学）；Siemens Industry Software NV（西门子工业软件公司）；Norwegian University of Science and Technology（挪威科技大学）
