Exploring to what humans pay attention in dynamic panoramic scenes is useful for many fundamental applications, including augmented reality (AR) in retail, AR-powered recruitment, and visual language navigation. With this goal in mind, we propose PV-SOD, a new task that aims to segment salient objects from panoramic videos. In contrast to existing fixation-/object-level saliency detection tasks, we focus on audio-induced salient object detection (SOD), where the salient objects are labeled with the guidance of audio-induced eye movements. To support this task, we collect the first large-scale dataset, named ASOD60K, which contains 4K-resolution video frames annotated with a six-level hierarchy, thus distinguishing itself with richness, diversity and quality. Specifically, each sequence is marked with both its super-/sub-class, with objects of each sub-class being further annotated with human eye fixations, bounding boxes, object-/instance-level masks, and associated attributes (e.g., geometrical distortion). These coarse-to-fine annotations enable detailed analysis for PV-SOD modelling, e.g., determining the major challenges for existing SOD models, and predicting scanpaths to study the long-term eye fixation behaviors of humans. We systematically benchmark 11 representative approaches on ASOD60K and derive several interesting findings. We hope this study could serve as a good starting point for advancing SOD research towards panoramic videos. The dataset and benchmark will be made publicly available at https://github.com/PanoAsh/ASOD60K.
Peking University Shenzhen Graduate School（北京大学深圳研究生院）
Almost all previous works on saliency detection have been dedicated to conventional images, however, with the outbreak of panoramic images due to the rapid development of VR or AR technology, it is becoming more challenging, meanwhile valuable for extracting salient contents in panoramic images. In this paper, we propose a novel bottom-up salient object detection framework for panoramic images. First, we employ a spatial density estimation method to roughly extract object proposal regions, with the help of region growing algorithm. Meanwhile, an eye fixation model is utilized to predict visual…
Video salient object detection (VSOD), as a fundamental computer vision problem, has been extensively discussed in the last decade. However, all existing works focus on addressing the VSOD problem in 2D scenarios. With the rapid development of VR devices, panoramic videos have been a promising alternative to 2D videos to provide immersive feelings of the real world. In this paper, we aim to tackle the video salient object detection problem for panoramic videos, with their corresponding ambisonic audios. A multimodal fusion module equipped with two pseudo-siamese audio-visual context fusion (AC…
Image-based salient object detection (ISOD) in 360° scenarios is significant for understanding and applying panoramic information. However, research on 360° ISOD has not been widely explored due to the lack of large, complex, high-resolution, and well-labeled datasets. Towards this end, we construct a large scale 360° ISOD dataset with object-level pixel-wise annotation on equirectangular projection (ERP), which contains rich panoramic scenes with not less than 2K resolution and is the largest dataset for 360° ISOD by far to our best knowledge. By observing the data, we find current methods fa…
Fixation prediction (FP) in panoramic contents has been widely investigated along with the booming trend of virtual reality (VR) applications. However, another issue within the field of visual saliency, salient object detection (SOD), has been seldom explored in 360° (or omnidirectional) images due to the lack of datasets representative of real scenes with pixel-level annotations. Toward this end, we collect 107 equirectangular panoramas with challenging scenes and multiple object classes. Based on the consistency between FP and explicit saliency judgements, we further manually annotate 1,165 …
Institut National des Sciences Appliquées de Rennes（雷恩国立应用科学学院）；Northeast Petroleum University（东北石油大学）
Salient human detection (SHD) in dynamic 360° immersive videos is of great importance for various applications such as robotics, inter-human and human-object interaction in augmented reality. However, 360° video SHD has been seldom discussed in the computer vision community due to a lack of datasets with large-scale omnidirectional videos and rich annotations. To this end, we propose SHD360, the first 360° video SHD dataset which contains various real-life daily scenes. Since so far there is no method proposed for 360° image/video SHD, we systematically benchmark 11 representative state-of-the…
Seoul National University（首尔大学）；Allen Institute for AI（艾伦人工智能研究所）；University of Oxford（牛津大学）；Hyundai Motor Company（现代汽车公司）
360$^\circ$ videos convey holistic views for the surroundings of a scene. It provides audio-visual cues beyond pre-determined normal field of views and displays distinctive spatial relations on a sphere. However, previous benchmark tasks for panoramic videos are still limited to evaluate the semantic understanding of audio-visual relationships or spherical spatial property in surroundings. We propose a novel benchmark named Pano-AVQA as a large-scale grounded audio-visual question answering dataset on panoramic videos. Using 5.4K 360$^\circ$ video clips harvested online, we collect two types o…
Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；McGill University（麦吉尔大学）
Audio-Visual scene understanding is a challenging problem due to the unstructured spatial-temporal relations that exist in the audio signals and spatial layouts of different objects and various texture patterns in the visual images. Recently, many studies have focused on abstracting features from convolutional neural networks while the learning of explicit semantically relevant frames of sound signals and visual images has been overlooked. To this end, we present an end-to-end framework, namely attentional graph convolutional network (AGCN), for structure-aware audio-visual scene representatio…
National Laboratory of Pattern Recognition, Institution of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所模式识别国家重点实验室）；School of Artificial Intelligence (AI), University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；CAS Center for Excellence in Brain Science and Intelligence Technology (CEBSIT)（中国科学院脑科学与智能技术卓越创新中心）；Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）；MarkableAI Inc.（MarkableAI公司）
Recently, video streams have occupied a large proportion of Internet traffic, most of which contain human faces. Hence, it is necessary to predict saliency on multiple-face videos, which can provide attention cues for many content based applications. However, most of multiple-face saliency prediction works only consider visual information and ignore audio, which is not consistent with the naturalistic scenarios. Several behavioral studies have established that sound influences human attention, especially during the speech turn-taking in multiple-face videos. In this paper, we thoroughly invest…
Human visual scene understanding is so remarkable that we are able to recognize a revisited place when entering it from the opposite direction it was first visited, even in the presence of extreme variations in appearance. This capability is especially apparent during driving: a human driver can recognize where they are when travelling in the reverse direction along a route for the first time, without having to turn back and look. The difficulty of this problem exceeds any addressed in past appearance- and viewpoint-invariant visual place recognition (VPR) research, in part because large parts…
Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics
University of Technology Sydney（悉尼科技大学）；The University of Queensland（昆士兰大学）；Futureverse；The Hong Kong University of Science and Technology（香港科技大学）；CSIRO DATA61（联邦科学与工业研究组织数据61研究所）；Netease Fuxi AI Lab（网易伏羲人工智能实验室）
UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions
State Key Laboratory for Novel Software Technology（新型软件技术国家重点实验室）；Tencent Hunyuan（腾讯文英）；Shanghai Jiao Tong University（上海交通大学）；Renmin University of China（中国人民大学）；Tsinghua University（清华大学）；Shanghai AI Lab（上海人工智能实验室）
Hunan University of Science and Technology（湖南科技大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Ocean University of China（中国海洋大学）
Content-based adult video detection plays an important role in preventing pornography. However, existing methods usually rely on single modality and seldom focus on multi-modality semantics representation. Addressing at this problem, we put forward an approach of analyzing periodicity and saliency for adult video detection. At first, periodic patterns and salient regions are respective-ly analyzed in audio-frames and visual-frames. Next, the multi-modal co-occurrence semantics is described by combining audio periodicity with visual saliency. Moreover, the performance of our approach is evaluat…
Thapar Institute of Engineering and Technology（塔帕尔工程技术学院）；Indian Institute of Technology Ropar（印度理工学院鲁尔基分校）；Oxford Brookes University（牛津布鲁克斯大学）
Recently, there has been numerous breakthroughs in face hallucination tasks. However, the task remains rather challenging in videos in comparison to the images due to inherent consistency issues. The presence of extra temporal dimension in video face hallucination makes it non-trivial to learn the facial motion through out the sequence. In order to learn these fine spatio-temporal motion details, we propose a novel cross-modal audio-visual Video Face Hallucination Generative Adversarial Network (VFH-GAN). The architecture exploits the semantic correlation of between the movement of the facial …
Columbia University（哥伦比亚大学）；Adobe Research（奥多比研究院）
Although 360\textdegree{} cameras ease the capture of panoramic footage, it remains challenging to add realistic 360\textdegree{} audio that blends into the captured scene and is synchronized with the camera motion. We present a method for adding scene-aware spatial audio to 360\textdegree{} videos in typical indoor scenes, using only a conventional mono-channel microphone and a speaker. We observe that the late reverberation of a room's impulse response is usually diffuse spatially and directionally. Exploiting this fact, we propose a method that synthesizes the directional impulse response b…
OmniAudio: Generating Spatial Audio from 360-Degree Video
Hong Kong University of Science and Technology（香港科技大学）；Tongyi Lab, Alibaba Group（阿里巴巴集团通义实验室）；Zhejiang University（浙江大学）；FAIR, Meta（Meta FAIR）；Nanyang Technological University（南洋理工大学）
Text-Audio-Visual-conditioned Diffusion Model for Video Saliency Prediction
School of Computer Science, Nanjing University of Information Science and Technology（信息科学与技术南京大学计算机科学学院）；Jiangsu Collaborative Innovation Center of Atmospheric Environment and Equipment Technology, Nanjing University of Information Science and Technology（大气环境与设备技术江苏省协同创新中心）；School of Computer Science and Informatics, Cardiff University（计算机科学与信息学院卡斯尔大学）；Faculty of Information Technology and Communication Sciences, Tampere University（信息科技与通讯科学学院坦佩雷大学）
AVTENet: A Human-Cognition-Inspired Audio-Visual Transformer-Based Ensemble Network for Video Deepfake Detection
Social Networks and Human-Centered Computing Program, Taiwan International Graduate Program, Academia Sinica（社会网络与以人为本计算计划、台湾国际研究生计划、中国科学院）；Institute of Information Systems and Applications, National Tsing Hua University（信息系统与应用研究所、国立清华大学）；Department of Computer Science, National Chengchi University（计算机科学系、国立成功大学）；Research Center for Information Technology Innovation, Academia Sinica（信息技术创新研究中心、中国科学院）；Institute of Information Science, Academia Sinica（信息科学研究所、中国科学院）
