AVS-Net: Audio-Visual Scale Net for Self-supervised Monocular Metric Depth Estimation
SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University（上海交通大学巴黎卓越工程师学院）；Center for Robotics, MINES Paris, PSL University（巴黎矿业学院机器人中心（巴黎文理研究大学））
Xiaohu Liu, Sascha Hornauer, Fabien Moutarde, Jialiang Lu
Metric depth prediction from monocular videos suffers from bad generalization between datasets and requires supervised depth data for scale-correct training. Self-supervised training using multi-view reconstruction can benefit from large scale natural videos but not provide correct scale, limiting its benefits. Recently, reflecting audible Echoes off objects is investigated for improved depth prediction and was shown to be sufficient to reconstruct objects at scale even without a visual signal. Because Echoes travel at fixed speed, they have the potential to resolve ambiguities in object scale and appearance. However, predicting depth end-to-end from sound and vision cannot benefit from unsupervised depth prediction approaches, which can process large scale data without sound annotation. In this work we show how Echoes can benefit depth prediction in two ways: When learning metric depth learned from supervised data and as supervisory signal for scale-correct self-supervised training. We show how we can improve the predictions of several state-of-the-art approaches and how the method can scale-correct a self-supervised depth approach.
Huawei Munich Research Center（华为慕尼黑研究中心）；Univ. Autònoma de Barcelona (UAB)（巴塞罗那自治大学）；Computer Vision Center (CVC)（计算机视觉中心）
Monocular Depth Estimation (MDE) is performed to produce 3D information that can be used in downstream tasks such as those related to on-board perception for Autonomous Vehicles (AVs) or driver assistance. Therefore, a relevant arising question is whether the standard metrics for MDE assessment are a good indicator of the accuracy of future MDE-based driving-related perception tasks. We address this question in this paper. In particular, we take the task of 3D object detection on point clouds as a proxy of on-board perception. We train and test state-of-the-art 3D object detectors using 3D poi…
AVS-Mamba: Exploring Temporal and Multi-modal Mamba for Audio-Visual Segmentation
School of Information and Communication Engineering, Dalian University of Technology（大连理工大学信息与通信工程学院）；School of Innovation and Entrepreneurship, Dalian University of Technology（大连理工大学创新创业学院）；School of Artificial Intelligence, Dalian University of Technology（大连理工大学人工智能学院）
Hefei University of Technology（合肥工业大学）；Sensetime Research（商汤科技研究院）；University of Oxford（牛津大学）；Beihang University（北京航空航天大学）；Australian National University（澳大利亚国立大学）；Nvidia（英伟达）；University of Hong Kong（香港大学）；Shanghai AI Lab（上海人工智能实验室）
We propose a new problem called audio-visual segmentation (AVS), in which the goal is to output a pixel-level map of the object(s) that produce sound at the time of the image frame. To facilitate this research, we construct the first audio-visual segmentation benchmark, i.e., AVSBench, providing pixel-wise annotations for sounding objects in audible videos. It contains three subsets: AVSBench-object (Single-source subset, Multi-sources subset) and AVSBench-semantic (Semantic-labels subset). Accordingly, three settings are studied: 1) semi-supervised audio-visual segmentation with a single soun…
CPM: Class-conditional Prompting Machine for Audio-visual Segmentation
Australian Institute for Machine Learning（澳大利亚机器学习研究院）；University of Adelaide（阿德莱德大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Surrey（萨里大学）
Georgia Tech（佐治亚理工学院）；Mitsubishi Electric Research Laboratories (MERL)（三菱电机研究实验室）
Scene-aware dialog systems will be able to have conversations with users about the objects and events around them. Progress on such systems can be made by integrating state-of-the-art technologies from multiple research areas including end-to-end dialog systems visual dialog, and video description. We introduce the Audio Visual Scene Aware Dialog (AVSD) challenge and dataset. In this challenge, which is one track of the 7th Dialog System Technology Challenges (DSTC7) workshop1, the task is to build a system that generates responses in a dialog about an input video
Ref-AVS: Refer and Segment Objects in Audio-Visual Scenes
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Beijing University of Posts and Telecommunications（北京邮电大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
University of California, Davis（加利福尼亚大学戴维斯分校）；Facebook AI Research (FAIR)（脸书人工智能研究院）
We present Audiovisual SlowFast Networks, an architecture for integrated audiovisual perception. AVSlowFast has Slow and Fast visual pathways that are deeply integrated with a Faster Audio pathway to model vision and sound in a unified representation. We fuse audio and visual features at multiple layers, enabling audio to contribute to the formation of hierarchical audiovisual concepts. To overcome training difficulties that arise from different learning dynamics for audio and visual modalities, we introduce DropPathway, which randomly drops the Audio pathway during training as an effective re…
Stepping Stones: A Progressive Training Strategy for Audio-Visual Semantic Segmentation
University of Chinese Academy of Sciences（中国科学院大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Engineering Research Center of Next-Generation Search and Recommendation（下一代搜索与推荐工程研究中心）
Audio-Visual Segmentation via Depth-Guided Collaborative Modeling
School of Intelligent Science and Technology, University of Science and Technology Beijing（北京科技大学智能科学与技术学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
When AVSR Meets Video Conferencing: Dataset, Degradation, and the Hidden Mechanism Behind Performance Collapse
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education（航空航天信息安全部与可信计算教育部重点实验室）；School of Cyber Science and Engineering, Wuhan University（武汉大学网络安全科学与工程学院）；Tsinghua University（清华大学）
NTT Computer and Data Science Laboratories, NTT Corporation（日本电信电话株式会社计算机与数据科学实验室）
There have been many attempts to build multimodal dialog systems that can respond to a question about given audio-visual information, and the representative task for such systems is the Audio Visual Scene-Aware Dialog (AVSD). Most conventional AVSD models adopt the Convolutional Neural Network (CNN)-based video feature extractor to understand visual information. While a CNN tends to obtain both temporally and spatially local information, global information is also crucial for boosting video understanding because AVSD requires long-term temporal visual dependency and whole visual information. I…
Audio-based automatic speech recognition (ASR) degrades significantly in noisy environments and is particularly vulnerable to interfering speech, as the model cannot determine which speaker to transcribe. Audio-visual speech recognition (AVSR) systems improve robustness by complementing the audio stream with the visual information that is invariant to noise and helps the model focus on the desired speaker. However, previous AVSR work focused solely on the supervised learning setup; hence the progress was hindered by the amount of labeled data available. In this work, we present a self-supervis…
University of the Chinese Academy of Sciences（中国科学院大学）
The Aduio-visual Speech Recognition (AVSR) which employs both the video and audio information to do Automatic Speech Recognition (ASR) is one of the application of multimodal leaning making ASR system more robust and accuracy. The traditional models usually treated AVSR as inference or projection but strict prior limits its ability. As the revival of deep learning, Deep Neural Networks (DNN) becomes an important toolkit in many traditional classification tasks including ASR, image classification, natural language processing. Some DNN models were used in AVSR like Multimodal Deep Autoencoders (…
Hefei University of Technology（合肥工业大学）；SenseTime Research（商汤科技研究院）；The University of Hong Kong（香港大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Australian National University（澳大利亚国立大学）；Beihang University（北京航空航天大学）；NVIDIA（英伟达）
We propose to explore a new problem called audio-visual segmentation (AVS), in which the goal is to output a pixel-level map of the object(s) that produce sound at the time of the image frame. To facilitate this research, we construct the first audio-visual segmentation benchmark (AVSBench), providing pixel-wise annotations for the sounding objects in audible videos. Two settings are studied with this benchmark: 1) semi-supervised audio-visual segmentation with a single sound source and 2) fully-supervised audio-visual segmentation with multiple sound sources. To deal with the AVS problem, we …
Can Textual Semantics Mitigate Sounding Object Segmentation Preference?
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Beijing University of Posts and Telecommunications（北京邮电大学）；University of Edinburgh（爱丁堡大学）
AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual Segmentation
School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；China Tower Corporation Limited（中国铁塔股份有限公司）
Audio-Visual Speech Enhancement In Complex Scenarios With Separation And Dereverberation Joint Modeling
School of Cyber Science and Engineering, Wuhan University（武汉大学计算机科学与工程学院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；Suzhou Municipal Key Laboratory of Multimodal Intelligent Systems, Digital Innovation Research Center, Duke Kunshan University（多模态智能系统苏州市级重点实验室、杜克昆山大学数字创新研究中心）；Hardware Engineering System, OPPO（OPPO硬件工程系统）
Multidisciplinary Institute of Artificial Intelligence (MIAI)（多学科人工智能研究所）
This paper investigates the impact of head movements on audio-visual speech enhancement (AVSE). Although being a common conversational feature, head movements have been ignored by past and recent studies: they challenge today's learning-based methods as they often degrade the performance of models that are trained on clean, frontal, and steady face images. To alleviate this problem, we propose to use robust face frontalization (RFF) in combination with an AVSE method based on a variational auto-encoder (VAE) model. We briefly describe the basic ingredients of the proposed pipeline and we perfo…
Discovering Sounding Objects by Audio Queries for Audio Visual Segmentation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机学院）；Alibaba Group（阿里巴巴集团）；Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）
