Ambisonics is a complete theory for spatial audio whose building blocks are the spherical harmonics. Some of the drawbacks of low order Ambisonics, like poor source directivity and small sweet-spot, are directly related to the properties of spherical harmonics. In this thesis we illustrate a novel spatial audio framework similar in spirit to Ambisonics that replaces the spherical harmonics by an alternative set of functions with compact support: the spherical wavelets. We develop a complete audio chain from encoding to decoding, using discrete spherical wavelets built on a multiresolution mesh. We show how the wavelet family and the decoding matrices to loudspeakers can be generated via numerical optimization. In particular, we present a decoding algorithm optimizing acoustic and psychoacoustic parameters that can generate decoding matrices to irregular layouts for both Ambisonics and the new wavelet format. This audio workflow is directly compared with Ambisonics.
Comparative Analysis of Mel-Frequency Cepstral Coefficients and Wavelet Based Audio Signal Processing for Emotion Detection and Mental Health Assessment in Spoken Speech
Dipartimento di Ingegneria Navale, Elettrica, Elettronica e delle Telecomunicazioni (DITEN)（海军、电气、电子与电信工程系）；Universidad EAFIT（EAFIT大学）
We propose a study of the mathematical properties of voice as an audio signal. This work includes signals in which the channel conditions are not ideal for emotion recognition. Multiresolution analysis discrete wavelet transform was performed through the use of Daubechies Wavelet Family (Db1-Haar, Db 6, Db8, Db10) allowing the decomposition of the initial audio signal into sets of coefficients on which a set of features was extracted and analyzed statistically in order to differentiate emotional states. ANNs proved to be a system that allows an appropriate classification of such states. This s…
We propose a time-domain audio source separation method using down-sampling (DS) and up-sampling (US) layers based on a discrete wavelet transform (DWT). The proposed method is based on one of the state-of-the-art deep neural networks, Wave-U-Net, which successively down-samples and up-samples feature maps. We find that this architecture resembles that of multiresolution analysis, and reveal that the DS layers of Wave-U-Net cause aliasing and may discard information useful for the separation. Although the effects of these problems may be reduced by training, to achieve a more reliable source s…
We introduce the joint time-frequency scattering transform, a time shift invariant descriptor of time-frequency structure for audio classification. It is obtained by applying a two-dimensional wavelet transform in time and log-frequency to a time-frequency wavelet scalogram. We show that this descriptor successfully characterizes complex time-frequency phenomena such as time-varying filters and frequency modulated excitations. State-of-the-art results are achieved for signal reconstruction and phone segment classification on the TIMIT dataset.
SALM: Spatial Audio Language Model with Structured Embeddings for Understanding and Editing
Institute of Acoustics, Chinese Academy of Sciences, China（中国科学院声学研究所）；MiLM Plus, Xiaomi Inc., China（小米公司）；Xi’an Jiaotong Liverpool University, China（西安交通大学利物浦大学）；University of Chinese Academy of Sciences, China（中国科学院大学）
WTFormer: A Wavelet Conformer Network for MIMO Speech Enhancement with Spatial Cues Peservation
Laboratory of Noise and Audio Research, Institute of Acoustics, Chinese Academy of Sciences（声学研究所噪声与音频研究实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey（塞夫尔大学视觉、语音与信号处理中心（CVSSP））
Learning from audio-visual data offers many possibilities to express correspondence between the audio and visual content, similar to the human perception that relates aural and visual information. In this work, we present a method for self-supervised representation learning based on audio-visual spatial alignment (AVSA), a more sophisticated alignment task than the audio-visual correspondence (AVC). In addition to the correspondence, AVSA also learns from the spatial location of acoustic and visual content. Based on 360$^\text{o}$ video and Ambisonics audio, we propose selection of visual obje…
Past, Present, and Future of Spatial Audio and Room Acoustics
National Institute of Informatics（情报信息研究所）；University of Surrey（萨里大学）；Australian National University（澳大利亚国立大学）；Dolby Laboratories（杜比实验室）；Politecnico di Milano（米兰理工大学）
Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK（视觉、语音和信号处理中心（CVSSP），英国萨里大学）；Tencent AI Lab, Bellevue, WA, USA（腾讯AI实验室，美国西雅图）；Department of Computer Science and Technology, University of Science and Technology Beijing, China（北京科技大学计算机科学与技术系）；The Chinese University of Hong Kong (CUHK)（香港中文大学）
School of Marine Science and Technology, Northwestern Polytechnical University（西北工业大学海洋科学与技术学院）；Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））；Research and Development Institute of Northwestern Polytechnical University in Shenzhen（西北工业大学深圳研究院）
Audio Spatially-Guided Fusion for Audio-Visual Navigation
Joint Research Laboratory for Embodied Intelligence, Xinjiang University（新疆大学联合研究实验室，具身智能）；Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University（新疆大学丝绸之路多语言认知计算国际联合研究实验室）；School of Computer Science and Technology, Xinjiang University（新疆大学计算机科学与技术学院）
SAIL: Spatial Audio Intelligence with Large Language Models via Disentangled Acoustic-Spatial Encoding and Dual-Stream Q-Former
Nanyang Technological University（南洋理工大学）；Singapore Management University（新加坡管理大学）；Tencent Hy Frontier Lab（腾讯Hy前沿实验室）；Beijing Institute of Technology（北京理工大学）；University of Surrey（萨里大学）
Self-supervised audio-visual learning aims to capture useful representations of video by leveraging correspondences between visual and audio inputs. Existing approaches have focused primarily on matching semantic information between the sensory streams. We propose a novel self-supervised task to leverage an orthogonal principle: matching spatial information in the audio stream to the positions of sound sources in the visual stream. Our approach is simple yet effective. We train a model to determine whether the left and right audio channels have been flipped, forcing it to reason about spatial …
OmniAudio: Generating Spatial Audio from 360-Degree Video
Hong Kong University of Science and Technology（香港科技大学）；Tongyi Lab, Alibaba Group（阿里巴巴集团通义实验室）；Zhejiang University（浙江大学）；FAIR, Meta（Meta FAIR）；Nanyang Technological University（南洋理工大学）
Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation
Hong Kong University of Science and Technology（香港科技大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Nanyang Technological University（南洋理工大学）；Zhejiang University（浙江大学）
National Taiwan University（台湾大学）；ASUS Intelligent Cloud Services（华硕智能云服务）
Human perceives rich auditory experience with distinct sound heard by ears. Videos recorded with binaural audio particular simulate how human receives ambient sound. However, a large number of videos are with monaural audio only, which would degrade the user experience due to the lack of ambient information. To address this issue, we propose an audio spatialization framework to convert a monaural video into a binaural one exploiting the relationship across audio and visual components. By preserving the left-right consistency in both audio and visual modalities, our learning strategy can be vie…
GRAM: Spatial general-purpose audio representation models for real-world applications
Donders Institute, Radboud University, Nijmegen, The Netherlands（多纳茨研究所，拉德堡德大学，尼姆韦根，荷兰）；Mortimer B Zuckerman Institute, Columbia University, New York, United States（莫蒂默·B·祖克erman研究所，哥伦比亚大学，纽约，美国）
Immersive audio-visual perception relies on the spatial integration of both auditory and visual information which are heterogeneous sensing modalities with different fields of reception and spatial resolution. This study investigates the perceived coherence of audiovisual object events presented either centrally or peripherally with horizontally aligned/misaligned sound. Various object events were selected to represent three acoustic feature classes. Subjective test results in a simulated virtual environment from 18 participants indicate a wider capture region in the periphery, with an outward…
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Department of Computer Science, The University of Texas at Dallas（德克萨斯大学达拉斯分校计算机科学系）
We introduce an approach to convert mono audio recorded by a 360 video camera into spatial audio, a representation of the distribution of sound over the full viewing sphere. Spatial audio is an important component of immersive 360 video viewing, but spatial audio microphones are still rare in current 360 video production. Our system consists of end-to-end trainable neural networks that separate individual sound sources and localize them on the viewing sphere, conditioned on multi-modal analysis of audio and 360 video frames. We introduce several datasets, including one filmed ourselves, and on…
National Institute of Advanced Industrial Science and Technology（日本产业技术综合研究所）；Waseda University（早稻田大学）
Detecting sound source objects within visual observation is important for autonomous robots to comprehend surrounding environments. Since sounding objects have a large variety with different appearances in our living environments, labeling all sounding objects is impossible in practice. This calls for self-supervised learning which does not require manual labeling. Most of conventional self-supervised learning uses monaural audio signals and images and cannot distinguish sound source objects having similar appearances due to poor spatial information in audio signals. To solve this problem, thi…
