We propose a self-supervised approach for learning to perform audio source separation in videos based on natural language queries, using only unlabeled video and audio pairs as training data. A key challenge in this task is learning to associate the linguistic description of a sound-emitting object to its visual features and the corresponding components of the audio waveform, all without access to annotations during training. To overcome this challenge, we adapt off-the-shelf vision-language foundation models to provide pseudo-target supervision via two novel loss functions and encourage a stronger alignment between the audio, visual and natural language modalities. During inference, our approach can separate sounds given text, video and audio input, or given text and audio input alone. We demonstrate the effectiveness of our self-supervised approach on three audio-visual separation datasets, including MUSIC, SOLOS and AudioSet, where we outperform state-of-the-art strongly supervised approaches despite not using object detectors or text labels during training.
IIIT Hyderabad（印度国际信息技术研究所海得拉巴分校）；New York University（纽约大学）
This paper presents MAST, a new model for Multimodal Abstractive Text Summarization that utilizes information from all three modalities -- text, audio and video -- in a multimodal video. Prior work on multimodal abstractive text summarization only utilized information from the text and video modalities. We examine the usefulness and challenges of deriving information from the audio modality and present a sequence-to-sequence trimodal hierarchical attention-based model that overcomes these challenges by letting the model pay more attention to the text modality. MAST outperforms the current stat…
The goal of audio captioning is to translate input audio into its description using natural language. One of the problems in audio captioning is the lack of training data due to the difficulty in collecting audio-caption pairs by crawling the web. In this study, to overcome this problem, we propose to use a pre-trained large-scale language model. Since an audio input cannot be directly inputted into such a language model, we utilize guidance captions retrieved from a training dataset based on similarities that may exist in different audio. Then, the caption of the audio input is generated by u…
ETRI（韩国电子通信研究院）；KAIST（韩国科学技术院）；University of Science and Technology（科学技术大学）
For human-like agents, including virtual avatars and social robots, making proper gestures while speaking is crucial in human--agent interaction. Co-speech gestures enhance interaction experiences and make the agents look alive. However, it is difficult to generate human-like gestures due to the lack of understanding of how people gesture. Data-driven approaches attempt to learn gesticulation skills from human demonstrations, but the ambiguous and individual nature of gestures hinders learning. In this paper, we present an automatic gesture generation model that uses the multimodal context of …
University of Wherever（无论何处大学）；Nazarbayev University（纳扎尔巴耶夫大学）
In this paper, we study an approach to multimodal person verification using audio, visual, and thermal modalities. The combination of audio and visual modalities has already been shown to be effective for robust person verification. From this perspective, we investigate the impact of further increasing the number of modalities by adding thermal images. In particular, we implemented unimodal, bimodal, and trimodal verification systems using state-of-the-art deep learning architectures and compared their performance under clean and noisy conditions. We also compared two popular fusion approaches…
In recent years, audio-visual event localization has attracted much attention. It's purpose is to detect the segment containing audio-visual events and recognize the event category from untrimmed videos. Existing methods use audio-guided visual attention to lead the model pay attention to the spatial area of the ongoing event, devoting to the correlation between audio and visual information but ignoring the correlation between audio and spatial motion. We propose a past and future motion extraction (pf-ME) module to mine the visual motion from videos ,embedded into the past and future motion g…
Audio-Visual Segmentation via Depth-Guided Collaborative Modeling
School of Intelligent Science and Technology, University of Science and Technology Beijing（北京科技大学智能科学与技术学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Monash University（莫纳什大学）；Indian Institute of Technology Ropar（印度理工学院罗帕尔分校）；University of Melbourne（墨尔本大学）
In challenging real-life conditions such as extreme head-pose, occlusions, and low-resolution images where the visual information fails to estimate visual attention/gaze direction, audio signals could provide important and complementary information. In this paper, we explore if audio-guided coarse head-pose can further enhance visual attention estimation performance for non-prolific faces. Since it is difficult to annotate audio signals for estimating the head-pose of the speaker, we use off-the-shelf state-of-the-art models to facilitate cross-modal weak-supervision. During the training phase…
Relevance-guided Audio Visual Fusion for Video Saliency Prediction
School of Computer Science, Nanjing University of Information Science and Technology（南京信息工程大学计算机科学学院）；Jiangsu Collaborative Innovation Center of Atmospheric Environment and Equipment Technology, Nanjing University of Information Science and Technology（南京信息工程大学江苏省大气环境与装备技术协同创新中心）；College of Computer and Technology, Nanjing University of Aeronautics and Astronautics（南京航空航天大学计算机科学与技术学院）
Audio-guided face reenactment aims to generate a photorealistic face that has matched facial expression with the input audio. However, current methods can only reenact a special person once the model is trained or need extra operations such as 3D rendering and image post-fusion on the premise of generating vivid faces. To solve the above challenge, we propose a novel \emph{R}eal-time \emph{A}udio-guided \emph{M}ulti-face reenactment approach named \emph{APB2FaceV2}, which can reenact different target faces among multiple persons with corresponding reference face and drive audio signal as input…
Sounding Video Generator: A Unified Framework for Text-guided Sounding Video Generation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Audio-guided face reenactment aims at generating photorealistic faces using audio information while maintaining the same facial movement as when speaking to a real person. However, existing methods can not generate vivid face images or only reenact low-resolution faces, which limits the application value. To solve those problems, we propose a novel deep neural network named APB2Face, which consists of GeometryPredictor and FaceReenactor modules. GeometryPredictor uses extra head pose and blink state signals as well as audio to predict the latent landmark geometry information, while FaceReenact…
Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment
The University of Queensland（昆士兰大学）；NetEase Fuxi AI Lab（网易伏羲人工智能实验室）；Matrix Verse AI（矩阵 Verse 人工智能公司）；CSIRO Data61（联邦科学与工业研究组织 Data61 部门）；Macau University of Science and Technology（澳门科技大学）
OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学先进交叉科学学院）；Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所中文信息处理实验室）；University of Limerick（利默里克大学）；CUHK, Shenzhen（香港中文大学（深圳））
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Department of Computer Science, The University of Texas at Dallas（德克萨斯大学达拉斯分校计算机科学系）
Stereo-Talker: Audio-driven 3D Human Synthesis with Prior-Guided Mixture-of-Experts
Department of Automation, Tsinghua University（自动化系，清华大学）；Bytedance Inc.（字节跳动公司）；State Key Laboratory of Media Convergence and Communication, Communication University of China（媒体融合与传播国家重点实验室，中国传媒大学）；School of Artificial Intelligence, Beijing Normal University（人工智能学院，北京师范大学）
JAVEDIT: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curation
Zhejiang University（浙江大学）；Tencent Youtu Lab（腾讯优图实验室）；Nanjing University（南京大学）；University of Auckland（奥克兰大学）；Fudan University（复旦大学）；National University of Singapore（新加坡国立大学）
Audio-Guided Visual Perception for Audio-Visual Navigation
School of Computer Science and Technology, Xinjiang University（新疆大学计算机科学与技术学院）；Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing（丝绸之路多语种认知计算国际联合实验室）；Tsinghua University（清华大学）；Tianjin University of Technology（天津工业大学）
Narrating For You: Prompt-guided Audio-visual Narrating Face Generation Employing Multi-entangled Latent Space
Machine Intelligence Group, Department of CS&IS, BITS Pilani, Hyderabad Campus, India（机器智能组，计算机科学与信息学系，比斯汉学院海得拉巴校区，印度）；Georgia Institute of Technology, USA（佐治亚理工学院，美国）
Stereophonic audio is an indispensable ingredient to enhance human auditory experience. Recent research has explored the usage of visual information as guidance to generate binaural or ambisonic audio from mono ones with stereo supervision. However, this fully supervised paradigm suffers from an inherent drawback: the recording of stereophonic audio usually requires delicate devices that are expensive for wide accessibility. To overcome this challenge, we propose to leverage the vastly available mono data to facilitate the generation of stereophonic audio. Our key observation is that the task …
Self supervised representation learning has recently attracted a lot of research interest for both the audio and visual modalities. However, most works typically focus on a particular modality or feature alone and there has been very limited work that studies the interaction between the two modalities for learning self supervised representations. We propose a framework for learning audio representations guided by the visual modality in the context of audiovisual speech. We employ a generative audio-to-video training scheme in which we animate a still image corresponding to a given audio clip a…
Korea University（高丽大学）；NVIDIA Corporation（英伟达公司）；KAIST（韩国科学技术院）
The recent success in StyleGAN demonstrates that pre-trained StyleGAN latent space is useful for realistic video generation. However, the generated motion in the video is usually not semantically meaningful due to the difficulty of determining the direction and magnitude in the StyleGAN latent space. In this paper, we propose a framework to generate realistic videos by leveraging multimodal (sound-image-text) embedding space. As sound provides the temporal contexts of the scene, our framework learns to generate a video that is semantically consistent with sound. First, our sound inversion modu…
University of Science and Technology of China（中国科学技术大学）
Automatic emotion recognition (AER) is a challenging task due to the abstract concept and multiple expressions of emotion. Although there is no consensus on a definition, human emotional states usually can be apperceived by auditory and visual systems. Inspired by this cognitive process in human beings, it's natural to simultaneously utilize audio and visual information in AER. However, most traditional fusion approaches only build a linear paradigm, such as feature concatenation and multi-system fusion, which hardly captures complex association between audio and video. In this paper, we intro…
Semantic Noise Reduction via Teacher-Guided Dual-Path Audio-Visual Representation Learning
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Objecteye Inc.（北京眼神科技有限公司）
Active speaker detection in videos addresses associating a source face, visible in the video frames, with the underlying speech in the audio modality. The two primary sources of information to derive such a speech-face relationship are i) visual activity and its interaction with the speech signal and ii) co-occurrences of speakers' identities across modalities in the form of face and speech. The two approaches have their limitations: the audio-visual activity models get confused with other frequently occurring vocal activities, such as laughing and chewing, while the speakers' identity-based m…
