In this paper, we propose SemanticAC, a semantics-assisted framework for Audio Classification to better leverage the semantic information. Unlike conventional audio classification methods that treat class labels as discrete vectors, we employ a language model to extract abundant semantics from labels and optimize the semantic consistency between audio signals and their labels. We verify that simple textual information from labels and advanced pretraining models enable more abundant semantic supervision for better performance. Specifically, we design a text encoder to capture the semantic information from the text extension of labels. Then we map the audio signals to align with the semantics of corresponding class labels via an audio encoder and a similarity calculation module so as to enforce the semantic consistency. Extensive experiments on two audio datasets, ESC-50 and US8K demonstrate that our proposed method consistently outperforms the compared audio classification methods.
Multimodal Laryngoscopic Video Analysis for Assisted Diagnosis of Vocal Fold Paralysis
School of Computer Science（计算机科学学院）；Suzhou Municipal Key Laboratory of Multimodal Intelligent Systems（多模态智能系统苏州市重点实验室）；Sun Yat-sen Memorial Hospital of Sun Yat-sen University（中山大学孙逸仙纪念医院）；School of Artificial Intelligent（人工智能学院）；Department of Otorhinolaryngology（耳鼻喉科部门）
Exploring Audio Cues for Enhanced Test-Time Video Model Adaptation
Artificial Intelligence Research Institute, Shenzhen MSU-BIT University and Guangdong-Hong Kong-Macao Joint Laboratory for Emotional Intelligence and Pervasive Computing（人工智能研究院、深圳MSU-BIT大学及粤港澳大湾区情感智能与普适计算联合实验室）；School of Software Engineering, South China University of Technology（华南理工大学软件学院）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）
Hearing from Silence: Reasoning Audio Descriptions from Silent Videos via Vision-Language Model
Chinese Academy of Sciences（中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent AI Lab（腾讯AI实验室）；Institute of Automation School of Artificial Intelligence（自动化研究所人工智能学院）
Audio commands are a preferred communication medium to keep inspectors in the loop of civil infrastructure inspection performed by a semi-autonomous drone. To understand job-specific commands from a group of heterogeneous and dynamic inspectors, a model must be developed cost-effectively for the group and easily adapted when the group changes. This paper is motivated to build a multi-tasking deep learning model that possesses a Share-Split-Collaborate architecture. This architecture allows the two classification tasks to share the feature extractor and then split subject-specific and keyword-s…
Hear Me, See Me, Understand Me: Audio-Visual Autism Behavior Recognition
The University of Texas at Dallas（德克萨斯大学达拉斯分校）；School of Behavioral and Brain Sciences, The University of Texas at Dallas（德克萨斯大学达拉斯分校行为与脑科学学院）；Brown University（布朗大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Harbin Engineering University（哈尔滨工程大学）；University of Technology Sydney（悉尼科技大学）；University of Surrey（萨里大学）
Automated audio captioning aims to describe audio data with captions using natural language. Existing methods often employ an encoder-decoder structure, where the attention-based decoder (e.g., Transformer decoder) is widely used and achieves state-of-the-art performance. Although this method effectively captures global information within audio data via the self-attention mechanism, it may ignore the event with short time duration, due to its limitation in capturing local information in an audio signal, leading to inaccurate prediction of captions. To address this issue, we propose a method us…
Audio compression-assisted feature extraction for voice replay attack detection
The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Huawei Technology（华为技术有限公司）；Huawei International（华为国际）
Exploring the User Experience of AI-Assisted Sound Searching Systems for Creative Workflows
Centre for Vision, Speech and Signal Processing, University of Surrey（视觉、语音和信号处理中心，萨里大学）；Research and Development, British Broadcasting Corporation（英国广播公司研发部）
Depth estimation enables a wide variety of 3D applications, such as robotics, autonomous driving, and virtual reality. Despite significant work in this area, it remains open how to enable accurate, low-cost, high-resolution, and large-range depth estimation. Inspired by the flash-to-bang phenomenon (i.e. hearing the thunder after seeing the lightning), this paper develops FBDepth, the first audio-visual depth estimation framework. It takes the difference between the time-of-flight (ToF) of the light and the sound to infer the sound source depth. FBDepth is the first to incorporate video and au…
University of Campinas（坎皮纳斯大学）；Federal University of Technology – Paraná（巴拉那联邦理工大学）
We propose a method that quantifies the importance, namely relevance, of audio segments for classification in weakly-labelled problems. It works by drawing information from a set of class-wise one-vs-all classifiers. By selecting the classifiers used in each specific classification problem, the relevance measure adapts to different user-defined viewpoints without requiring additional neural network training. This characteristic allows the relevance measure to highlight audio segments that quickly adapt to user-defined criteria. Such functionality can be used for computer-assisted audio analysi…
Institut de Recherche et Coordination Acoustique-Musique (IRCAM)（声学与音乐研究协作研究所（IRCAM））；CNRS（法国国家科学研究中心）；UPMC - Sorbonne Universite（巴黎第六大学-索邦大学）
Generative models have thrived in computer vision, enabling unprecedented image processes. Yet the results in audio remain less advanced. Our project targets real-time sound synthesis from a reduced set of high-level parameters, including semantic controls that can be adapted to different sound libraries and specific tags. These generative variables should allow expressive modulations of target musical qualities and continuously mix into new styles. To this extent we train AEs on an orchestral database of individual note samples, along with their intrinsic attributes: note class, timbre domain…
FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Microsoft（微软公司）；Shanghai Jiao Tong University（上海交通大学）；Shenzhen Loop Area Institute（深圳河套学院）；City University of Macau（澳门城市大学）；Amphion Technology Co., Ltd.（Amphion科技有限公司）
