We propose a two-stream convolutional network for audio recognition, that operates on time-frequency spectrogram inputs. Following similar success in visual recognition, we learn Slow-Fast auditory streams with separable convolutions and multi-level lateral connections. The Slow pathway has high channel capacity while the Fast pathway operates at a fine-grained temporal resolution. We showcase the importance of our two-stream proposal on two diverse datasets: VGG-Sound and EPIC-KITCHENS-100, and achieve state-of-the-art results on both.
Traffic Pumping attacks are a form of high-volume SPAM that target telephone networks, defraud customers and squander telephony resources. One type of call in these attacks is characterized by very low-amplitude signal levels, notably below the auditory threshold. We propose a technique to classify so-called "dead air" or "silent" SPAM calls based on features derived from factorizing the caller audio spectrogram. We describe the algorithms for feature extraction and classification as well as our data collection methods and production performance on millions of calls per week.
Reverse the auditory processing pathway: Coarse-to-fine audio reconstruction from fMRI
Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology（脑认知与类脑智能技术重点实验室）；State Key Laboratory of Multimodal Artificial Intelligence Systems（多模态人工智能系统国家重点实验室）；School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）
Neural Text to Articulate Talk: Deep Text to Audiovisual Speech Synthesis achieving both Auditory and Photo-realism
National Technical University of Athens（雅典国家技术大学）；Athena Research Center（雅典娜研究中心）；Foundation for Research & Technology - Hellas (FORTH)（希腊研究与技术基金会（FORTH））
Source separation is the task to separate an audio recording into individual sound sources. Source separation is fundamental for computational auditory scene analysis. Previous work on source separation has focused on separating particular sound classes such as speech and music. Many of previous work require mixture and clean source pairs for training. In this work, we propose a source separation framework trained with weakly labelled data. Weakly labelled data only contains the tags of an audio clip, without the occurrence time of sound events. We first train a sound event detection system wi…
University of Pittsburgh（匹兹堡大学）；Carnegie Mellon University（卡内基梅隆大学）；Universidad de Sevilla（塞维利亚大学）
In recent years tremendous efforts have been done to advance the state of the art for Natural Language Processing (NLP) and audio recognition. However, these efforts often translated in increased power consumption and memory requirements for bigger and more complex models. These solutions falls short of the constraints of IoT devices which need low power, low memory efficient computation, and therefore they fail to meet the growing demand of efficient edge computing. Neuromorphic systems have proved to be excellent candidates for low-power low-latency computation in a multitude of applications…
Texas State University（德克萨斯州立大学）；University of Trento（特伦托大学）；EPFL（洛桑联邦理工学院）
The major challenge in audio-visual event localization task lies in how to fuse information from multiple modalities effectively. Recent works have shown that attention mechanism is beneficial to the fusion process. In this paper, we propose a novel joint attention mechanism with multimodal fusion methods for audio-visual event localization. Particularly, we present a concise yet valid architecture that effectively learns representations from multiple modalities in a joint manner. Initially, visual features are combined with auditory features and then turned into joint representations. Next, w…
RAIL: Rethinking Auditory Intelligence in Large Audio-Language Models with a CHC-Grounded Benchmark
School of Computing and Information Systems, The University of Melbourne（墨尔本大学计算与信息系统学院）；Faculty of Psychology and Educational Sciences, Alexandru Ioan Cuza University of Iași（亚历山德鲁伊万库扎大学心理学与教育科学学院）；School of Electronic Information, Wuhan University（武汉大学电子信息学院）；School of Public Health, The University of Hong Kong（香港大学公共卫生学院）；School of Computer Science, The University of Auckland（奥克兰大学计算机科学学院）；Department of Data Science and Artificial Intelligence, Monash University（莫纳什大学数据科学与人工智能系）
Better audio representations are more brain-like: linking model-brain alignment with performance in downstream auditory tasks
Instituto de Investigación en Ciencias de la Computación (ICC)（计算科学研究所（ICC））；CONICET-Universidad de Buenos Aires（CONICET-阿根廷布宜诺斯艾利斯大学）；Departamento de Computación, FCEyN（计算机系，FCEyN）；Universidad de Buenos Aires (UBA)（阿根廷布宜诺斯艾利斯大学（UBA））
This study describes a binaural machine hearing system that is capable of performing auditory stream segregation in scenarios where multiple sound sources are present. The process of stream segregation refers to the capability of human listeners to group acoustic signals into sets of distinct auditory streams, corresponding to individual sound sources. The proposed computational framework mimics this ability via a probabilistic clustering scheme for joint localization and segregation. This scheme is based on mixtures of von Mises distributions to model the angular positions of the sound source…
Audio-Visual Target Speaker Extraction with Reverse Selective Auditory Attention
National University of Singapore（新加坡国立大学）；University of Science and Technology Beijing（北京科技大学）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Shenzhen Research Institute of Big data（深圳大数据研究院）；University of Bremen（不来梅大学）
China University of Petroleum (East China)（中国石油大学（华东））；Shandong University of Finance and Economics（山东财经大学）；Beijing Information Science and Technology University（北京信息科技大学）
Video saliency detection (VSD) aims at fast locating the most attractive objects/things/patterns in a given video clip. Existing VSD-related works have mainly relied on the visual system but paid less attention to the audio aspect, while, actually, our audio system is the most vital complementary part to our visual system. Also, audio-visual saliency detection (AVSD), one of the most representative research topics for mimicking human perceptual mechanisms, is currently in its infancy, and none of the existing survey papers have touched on it, especially from the perspective of saliency detecti…
Amirkabir University of Technology（阿米尔卡比尔理工大学）；Alzahra University（阿尔扎赫拉大学）
visual information can be converted into audio stream via sensory substitution devices in order to give visually impaired people the chance of perception of their surrounding easily and simultaneous to performing everyday tasks. In this study, visual environmental features namely, coordinate, type of objects and their size are assigned to audio features related to music tones such as frequency, time duration and note permutations. Results demonstrated that this new method has more training time efficiency in comparison with our previous method named VBTones which sinusoidal tones were applied.…
Department of Brain and Cognitive Sciences, MIT, USA（脑科学与认知科学系，麻省理工学院）；McGovern Institute for Brain Research, MIT, USA（脑研究麦戈文研究所，麻省理工学院）；Center for Brains, Minds and Machines, MIT, USA（脑、心智与机器中心，麻省理工学院）；MIT-IBM Watson AI Lab, IBM Research, USA（麻省理工-IBM沃森人工智能实验室，IBM研究院）
Separating an audio scene such as a cocktail party into constituent, meaningful components is a core task in computer audition. Deep networks are the state-of-the-art approach. They are trained on synthetic mixtures of audio made from isolated sound source recordings so that ground truth for the separation is known. However, the vast majority of available audio is not isolated. The brain uses primitive cues that are independent of the characteristics of any particular sound source to perform an initial segmentation of the audio scene. We present a method for bootstrapping a deep model for musi…
Lightly Weighted Automatic Audio Parameter Extraction for the Quality Assessment of Consensus Auditory-Perceptual Evaluation of Voice
National Taiwan University（台湾大学）；Research Center for Information Technology Innovation, Academia Sinica（中央研究院资讯科技创新研究中心）；National Taiwan University Hospital（台湾大学医院）；National Taiwan University College of Medicine（台湾大学医学院）；National Taiwan University Hospital Hsin-Chu Branch（台湾大学医院新竹分院）
Hijacking Large Audio-Language Models via Context-Agnostic and Imperceptible Auditory Prompt Injection
The State Key Laboratory of Blockchain and Data Security, Zhejiang University（区块链与数据安全国家重点实验室，浙江大学）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）；Nanyang Technological University（南洋理工大学）；National University of Singapore（新加坡国立大学）
In daily life, social interaction and acoustic communication often take place in complex acoustic environments (CAE) with a variety of interfering sounds and reverberation. For hearing research and the evaluation of hearing systems, simulated CAEs using virtual reality techniques have gained interest in the context of ecological validity. In the current study, the effect of scene complexity and visual representation of the scene on psychoacoustic measures like sound source location, distance perception, loudness, speech intelligibility, and listening effort in a virtual audio-visual environmen…
From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs
Surrey Institute for People-Centred AI (PAI)（萨里人本人工智能研究所）；University of Surrey（萨里大学）；Centre for Vision, Speech and Signal Processing (CVSSP)（视觉、语音和信号处理中心）
Audio-Guided Visual Perception for Audio-Visual Navigation
School of Computer Science and Technology, Xinjiang University（新疆大学计算机科学与技术学院）；Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing（丝绸之路多语种认知计算国际联合实验室）；Tsinghua University（清华大学）；Tianjin University of Technology（天津工业大学）
École de Technologie Supérieure, University of Québec（魁北克大学高等技术学院）
In this paper we propose a fusion approach to continuous emotion recognition that combines visual and auditory modalities in their representation spaces to predict the arousal and valence levels. The proposed approach employs a pre-trained convolution neural network and transfer learning to extract features from video frames that capture the emotional content. For the auditory content, a minimalistic set of parameters such as prosodic, excitation, vocal tract, and spectral descriptors are used as features. The fusion of these two modalities is carried out at a feature level, before training a …
Loss functions incorporating auditory spatial perception in deep learning -- a review
School of Electrical and Computer Engineering, Ben-Gurion University of the Negev（本-古里安大学电气与计算机工程学院）；Audio Communication Group, Technische Universität Berlin（柏林工业大学音频通信组）
CoSTALA: Compositional Spatio-Temporal Audio-Language Alignment via Multi-Grain Hierarchical Contrastive Learning
Xi’an Jiaotong Liverpool University（西交利物浦大学）；MiLM Plus, Xiaomi Inc.（小米公司MiLM Plus）；University of Oulu（奥卢大学）；Institute of Acoustics, Chinese Academy of Sciences（中国科学院声学研究所）
Jadavpur University（贾达普大学）；IIT Kharagpur（印度理工学院卡哈拉格普尔分校）；Shrutinandan School of Music（Shrutinandan音乐学院）
It is already known that both auditory and visual stimulus is able to convey emotions in human mind to different extent. The strength or intensity of the emotional arousal vary depending on the type of stimulus chosen. In this study, we try to investigate the emotional arousal in a cross-modal scenario involving both auditory and visual stimulus while studying their source characteristics. A robust fractal analytic technique called Detrended Fluctuation Analysis (DFA) and its 2D analogue has been used to characterize three (3) standardized audio and video signals quantifying their scaling expo…
INDIAN INSTITUTE OF SCIENCE EDUCATION AND RESEARCH, BHOPAL（博帕尔印度科学教育与研究所）
The approach used not only challenges some of the fundamental mathematical techniques used so far in early experiments of the same trend but also introduces new scopes and new horizons for interesting results. The physics governing spectrograms have been optimized in the project along with exploring how it handles the intense requirements of the problem at hand. Major contributions and developments brought under the light, through this project involve using better mathematical techniques and problem-specific machine learning methods. Improvised data analysis and data augmentation for audio dat…
Laboratoire BMBI Compiègne（贡比涅BMBI实验室）；Université de Technologie de Compiègne（贡比涅技术大学）
Machine hearing is an emerging area. Motivated by the need of a principled framework across domain applications for machine listening, we propose a generic and data-driven representation learning approach. For this sake, a novel and efficient supervised dictionary learning method is presented. Experiments are performed on both computational auditory scene (East Anglia and Rouen) and synthetic music chord recognition datasets. Obtained results show that our method is capable to reach state-of-the-art hand-crafted features for both applications
Vrije University of Brussels（布鲁塞尔自由大学）；Interuniversity Microelectronics Centre（校际微电子中心）；University of Bristol（布里斯托大学）
A key function of auditory cognition is the association of characteristic sounds with their corresponding semantics over time. Humans attempting to discriminate between fine-grained audio categories, often replay the same discriminative sounds to increase their prediction confidence. We propose an end-to-end attention-based architecture that through selective repetition attends over the most discriminative sounds across the audio sequence. Our model initially uses the full audio sequence and iteratively refines the temporal segments replayed based on slot attention. At each playback, the selec…
