With the rapid development of deep learning techniques, image saliency deep models trained solely by spatial information have occasionally achieved detection performance for video data comparable to that of the models trained by both spatial and temporal information. However, due to the lesser consideration of temporal information, the image saliency deep models may become fragile in the video sequences dominated by temporal information. Thus, the most recent video saliency detection approaches have adopted the network architecture starting with a spatial deep model that is followed by an elaborately designed temporal deep model. However, such methods easily encounter the performance bottleneck arising from the single stream learning methodology, so the overall detection performance is largely determined by the spatial deep model. In sharp contrast to the current mainstream methods, this paper proposes a novel plug-and-play scheme to weakly retrain a pretrained image saliency deep model for video data by using the newly sensed and coded temporal information. Thus, the retrained image saliency deep model will be able to maintain temporal saliency awareness, achieving much improved detection performance. Moreover, our method is simple yet effective for adapting any off-the-shelf pre-trained image saliency deep model to obtain high-quality video saliency detection. Additionally, both the data and source code of our method are publicly available.
Visual saliency modeling for images and videos is treated as two independent tasks in recent computer vision literature. While image saliency modeling is a well-studied problem and progress on benchmarks like SALICON and MIT300 is slowing, video saliency models have shown rapid gains on the recent DHF1K benchmark. Here, we take a step back and ask: Can image and video saliency modeling be approached via a unified model, with mutual benefit? We identify different sources of domain shift between image and video saliency data and between different video saliency datasets as a key challenge for ef…
360$^\circ$ video saliency detection is one of the challenging benchmarks for 360$^\circ$ video understanding since non-negligible distortion and discontinuity occur in the projection of any format of 360$^\circ$ videos, and capture-worthy viewpoint in the omnidirectional sphere is ambiguous by nature. We present a new framework named Panoramic Vision Transformer (PAVER). We design the encoder using Vision Transformer with deformable convolution, which enables us not only to plug pretrained models from normal videos into our architecture without additional modules or finetuning but also to per…
Universitat Politecnica de Catalunya（加泰罗尼亚理工大学）；Dublin City University（都柏林城市大学）
This work adapts a deep neural model for image saliency prediction to the temporal domain of egocentric video. We compute the saliency map for each video frame, firstly with an off-the-shelf model trained from static images, secondly by adding a a convolutional or conv-LSTM layers trained with a dataset for video saliency prediction. We study each configuration on EgoMon, a new dataset made of seven egocentric videos recorded by three subjects in both free-viewing and task-driven set ups. Our results indicate that the temporal adaptation is beneficial when the viewer is not moving and observin…
Shanghai Jiao Tong University（上海交通大学）；University of the West of England（西英格兰大学）；University College London（伦敦大学学院）
Joint saliency map (JSM) [1] was developed to assign high joint saliency values to the corresponding saliency structures (called Joint Saliency Structures, JSSs) but zero or low joint saliency values to the outliers (or mismatches) that are introduced by missing correspondence or local large deformations between the reference and moving images to be registered. JSM guides the local structure matching in nonrigid registration by emphasizing these JSSs' sparse deformation vectors in adaptive kernel regression of hierarchical sparse deformation vectors for iterative dense deformation reconstructi…
CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System, University of Science and Technology of China（中国科学技术大学 中国科学院空间信息处理与应用系统技术重点实验室）；KDDI Research（KDDI研究所）
Accelerating the inference speed of CNNs is critical to their deployment in real-world applications. Among all the pruning approaches, those implementing a sparsity learning framework have shown to be effective as they learn and prune the models in an end-to-end data-driven manner. However, these works impose the same sparsity regularization on all filters indiscriminately, which can hardly result in an optimal structure-sparse network. In this paper, we propose a Saliency-Adaptive Sparsity Learning (SASL) approach for further optimization. A novel and effective estimation of each filter, i.e.…
Dalian University of Technology, China（大连理工大学（中国））；Deepwise AI Lab, China（深度智谷人工智能实验室（中国））
The high cost of pixel-level annotations makes it appealing to train saliency detection models with weak supervision. However, a single weak supervision source usually does not contain enough information to train a well-performing model. To this end, we propose a unified framework to train saliency detection models with diverse weak supervision sources. In this paper, we use category labels, captions, and unlabelled data for training, yet other supervision sources can also be plugged into this flexible framework. We design a classification network (CNet) and a caption generation network (PNet)…
Duke University（杜克大学）；Duke University Medical Center（杜克大学医学中心）
Salient object detection (SOD) is viewed as a pixel-wise saliency modeling task by traditional deep learning-based methods. A limitation of current SOD models is insufficient utilization of inter-pixel information, which usually results in imperfect segmentation near edge regions and low spatial coherence. As we demonstrate, using a saliency mask as the only label is suboptimal. To address this limitation, we propose a connectivity-based approach called bilateral connectivity network (BiconNet), which uses connectivity masks together with saliency masks as labels for effective modeling of inte…
Northeast Petroleum University（东北石油大学）；University of Shanghai for Science and Technology（上海理工大学）；Inception Institute of Artificial Intelligence（起源人工智能研究院）；Heilongjiang University（黑龙江大学）
Saliency detection based on the complementary information from RGB images and depth maps has recently gained great popularity. In this paper, we propose Complementary Attention and Adaptive Integration Network (CAAI-Net), a novel RGB-D saliency detection model that integrates complementary attention based feature concentration and adaptive cross-modal feature fusion into a unified framework for accurate saliency detection. Specifically, we propose a context-aware complementary attention (CCA) module, which consists of a feature interaction component, a complementary attention component, and a …
Video classification is productive in many practical applications, and the recent deep learning has greatly improved its accuracy. However, existing works often model video frames indiscriminately, but from the view of motion, video frames can be decomposed into salient and non-salient areas naturally. Salient and non-salient areas should be modeled with different networks, for the former present both appearance and motion information, and the latter present static background information. To address this problem, in this paper, video saliency is predicted by optical flow without supervision fi…
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（北京航空航天大学虚拟现实技术与系统国家重点实验室）；Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）；AI Lab of China Merchants Bank（招商银行人工智能实验室）
Co-saliency detection aims to detect common salient objects from a group of relevant images. Some attempts have been made with the Fully Convolutional Network (FCN) framework and achieve satisfactory detection results. However, due to stacking convolution layers and pooling operation, the boundary details tend to be lost. In addition, existing models often utilize the extracted features without discrimination, leading to redundancy in representation since actually not all features are helpful to the final prediction and some even bring distraction. In this paper, we propose a co-attention modu…
College of Information Science & Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）
RGB-D salient object detection aims to identify the most visually distinctive objects in a pair of color and depth images. Based upon an observation that most of the salient objects may stand out at least in one modality, this paper proposes an adaptive fusion scheme to fuse saliency predictions generated from two modalities. Specifically, we design a two-streamed convolutional neural network (CNN), each of which extracts features and predicts a saliency map from either RGB or depth modality. Then, a saliency fusion module learns a switch map that is used to adaptively fuse the predicted salie…
High dynamic range (HDR) imaging enables to immortalize natural scenes similar to the way that they are perceived by human observers. With regular low dynamic range (LDR) capture/display devices, significant details may not be preserved in images due to the huge dynamic range of natural scenes. To minimize the information loss and produce high quality HDR-like images for LDR screens, this study proposes an efficient multi-exposure fusion (MEF) approach with a simple yet effective weight extraction method relying on principal component analysis, adaptive well-exposedness and saliency maps. Thes…
Nanjing University of Information Science and Technology（南京信息工程大学）；JD Digits（京东数科）
Co-saliency detection aims to discover the common and salient foregrounds from a group of relevant images. For this task, we present a novel adaptive graph convolutional network with attention graph clustering (GCAGC). Three major contributions have been made, and are experimentally shown to have substantial practical merits. First, we propose a graph convolutional network design to extract information cues to characterize the intra- and interimage correspondence. Second, we develop an attention graph clustering algorithm to discriminate the common objects from all the salient foreground objec…
Adaptive 3D Gaussian Splatting Video Streaming: Visual Saliency-Aware Tiling and Meta-Learning-Based Bitrate Adaptation
School of Electrical Engineering and Automation, Hefei University of Technology（电子工程与自动化学院，合肥工业大学）；Engineering Technology Research Center of Industrial Automation of Anhui Province（安徽省工业自动化工程技术研究中心）；School of Computer Science and Information Engineering, Hefei University of Technology（计算机科学与信息工程学院，合肥工业大学）；The University of Electro-Communications（电子通信大学）
Locating and Mitigating Gradient Conflicts in Point Cloud Domain Adaptation via Saliency Map Skewness
National Institute of Health Data Science, Peking University（北京大学国家健康数据科学研究院）；Institute of Medical Technology, Peking University（北京大学医学技术研究院）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）
Building Information Modeling (BIM) is a recent construction process based on a 3D model, containing every component related to the building achievement. Architects, structure engineers, method engineers, and others participant to the building process work on this model through the design-to-construction cycle. The high complexity and the large amount of information included in these models raise several issues, delaying its wide adoption in the industrial world. One of the most important is the visualization: professionals have difficulties to find out the relevant information for their job. …
STMicrolectronics, ADG Central R&D, Catania, Italy（意法半导体，ADG中央研发部，卡塔尼亚，意大利）；PeRCeiVe Lab - University of Catania（PeRCeiVe实验室 - 卡塔尼亚大学）
In this work, we propose a 3D fully convolutional architecture for video saliency prediction that employs hierarchical supervision on intermediate maps (referred to as conspicuity maps) generated using features extracted at different abstraction levels. We provide the base hierarchical learning mechanism with two techniques for domain adaptation and domain-specific learning. For the former, we encourage the model to unsupervisedly learn hierarchical general features using gradient reversal at multiple scales, to enhance generalization capabilities on datasets for which no annotations are provi…
Shanghai Jiao Tong University（上海交通大学）；University College London（伦敦大学学院）；Shanghai Renji Hospital, School of Medicine, Shanghai Jiaotong University（上海交通大学医学院附属上海仁济医院）；University of the West of England（西英格兰大学）
For nonrigid image registration, matching the particular structures (or the outliers) that have missing correspondence and/or local large deformations, can be more difficult than matching the common structures with small deformations in the two images. Most existing works depend heavily on the outlier segmentation to remove the outlier effect in the registration. Moreover, these works do not handle simultaneously the missing correspondences and local large deformations. In this paper, we defined the nonrigid image registration as a local adaptive kernel regression which locally reconstruct the…
University of British Columbia（不列颠哥伦比亚大学）；Huawei（华为）
The opacity of deep learning models constrains their debugging and improvement. Augmenting deep models with saliency-based strategies, such as attention, has been claimed to help get a better understanding of the decision-making process of black-box models. However, some recent works challenged saliency's faithfulness in the field of Natural Language Processing (NLP), questioning attention weights' adherence to the true decision-making process of the model. We add to this discussion by evaluating the faithfulness of in-model saliency applied to a video processing task for the first time, namel…
ViSAGE @ NTIRE 2026 Challenge on Video Saliency Prediction
Shandong University（山东大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；City University of Hong Kong（香港城市大学）；Shenzhen Loop Area Institute（深圳河套学院）
Predicting saliency in videos is a challenging problem due to complex modeling of interactions between spatial and temporal information, especially when ever-changing, dynamic nature of videos is considered. Recently, researchers have proposed large-scale datasets and models that take advantage of deep learning as a way to understand what's important for video saliency. These approaches, however, learn to combine spatial and temporal features in a static manner and do not adapt themselves much to the changes in the video content. In this paper, we introduce Gated Fusion Network for dynamic sal…
EEG-Driven Image Reconstruction with Saliency-Guided Diffusion Models
Ivannikov Institute for System Programming of the Russian Academy of Sciences（俄罗斯科学院伊万诺夫系统编程研究所）；Research Center for Trusted Artificial Intelligence（可信人工智能研究中心）；AI Talent Hub, ITMO University（ITMO大学人工智能人才中心）；Moscow, Russia（俄罗斯莫斯科）
Inception Institute of Artificial Intelligence（英赛普申人工智能研究院）
In many real world medical image classification settings we do not have access to samples of all possible disease classes, while a robust system is expected to give high performance in recognizing novel test data. We propose a generalized zero shot learning (GZSL) method that uses self supervised learning (SSL) for: 1) selecting anchor vectors of different disease classes; and 2) training a feature generator. Our approach does not require class attribute vectors which are available for natural images but not for medical images. SSL ensures that the anchor vectors are representative of each cla…
This paper proposes a novel saliency detection method by combining region-level saliency estimation and pixel-level saliency prediction with CNNs (denoted as CRPSD). For pixel-level saliency prediction, a fully convolutional neural network (called pixel-level CNN) is constructed by modifying the VGGNet architecture to perform multi-scale feature learning, based on which an image-to-image prediction is conducted to accomplish the pixel-level saliency detection. For region-level saliency estimation, an adaptive superpixel based region generation technique is first designed to partition an image …
Learning to Incorporate Texture Saliency Adaptive Attention to Image Cartoonization
University of Chinese Academy of Sciences（中国科学院大学）；Chinese Academy of Sciences（中国科学院）；Research Center on Fictitious Economy and Data Science（虚拟经济与数据科学研究中心）
As an important and challenging problem in computer vision, video saliency detection is typically cast as a spatiotemporal context modeling problem over consecutive frames. As a result, a key issue in video saliency detection is how to effectively capture the intrinsical properties of atomic video structures as well as their associated contextual interactions along the spatial and temporal dimensions. Motivated by this observation, we propose a graph-theoretic video saliency detection approach based on adaptive video structure discovery, which is carried out within a spatiotemporal atomic grap…
The prediction of Visual Attention data from any kind of media is of valuable use to content creators and used to efficiently drive encoding algorithms. With the current trend in the Virtual Reality (VR) field, adapting known techniques to this new kind of media is starting to gain momentum. In this paper, we present an architectural extension to any Convolutional Neural Network (CNN) to fine-tune traditional 2D saliency prediction to Omnidirectional Images (ODIs) in an end-to-end manner. We show that each step in the proposed pipeline works towards making the generated saliency map more accur…
Saliency-Guided Domain Adaptation for Left-Hand Driving in Autonomous Steering
Centre for Accident Research and Road Safety-Queensland, Queensland University of Technology（昆士兰事故研究与道路安全中心、昆士兰理工大学）；QUT Centre for Robotics and School of Electrical Engineering and Computer Science, Queensland University of Technology（昆士兰大学机器人中心及电气工程与计算机科学学院、昆士兰理工大学）
Indian Institute of Technology Gandhinagar（甘地纳格尔印度理工学院）
Saliency detection has drawn a lot of attention of researchers in various fields over the past several years. Saliency is the perceptual quality that makes an object, person to draw the attention of humans at the very sight. Salient object detection in an image has been used centrally in many computational photography and computer vision applications like video compression, object recognition and classification, object segmentation, adaptive content delivery, motion detection, content aware resizing, camouflage images and change blindness images to name a few. We propose a method to detect sal…
Universitat Politecnica de Catalunya（加泰罗尼亚理工大学）；Insight Center for Data Analytics（洞察数据分析中心）；Dublin City University（都柏林城市大学）
The prediction of salient areas in images has been traditionally addressed with hand-crafted features based on neuroscience principles. This paper, however, addresses the problem with a completely data-driven approach by training a convolutional neural network (convnet). The learning process is formulated as a minimization of a loss function that measures the Euclidean distance of the predicted saliency map with the provided ground truth. The recent publication of large datasets of saliency prediction has provided enough data to train end-to-end architectures that are both fast and accurate. T…
As an emerging vision platform, a drone can look from many abnormal viewpoints which brings many new challenges into the classic vision task of video saliency prediction. To investigate these challenges, this paper proposes a large-scale video dataset for aerial saliency prediction, which consists of ground-truth salient object regions of 1,000 aerial videos, annotated by 24 subjects. To the best of our knowledge, it is the first large-scale video dataset that focuses on visual saliency prediction on drones. Based on this dataset, we propose a Model-guided Multi-path Network (MM-Net) that serv…
This paper introduces a new framework to predict visual attention of omnidirectional images. The key setup of our architecture is the simultaneous prediction of the saliency map and a corresponding scanpath for a given stimulus. The framework implements a fully encoder-decoder convolutional neural network augmented by an attention module to generate representative saliency maps. In addition, an auxiliary network is employed to generate probable viewport center fixation points through the SoftArgMax function. The latter allows to derive fixation points from feature maps. To take advantage of th…
SOKENDAI (Graduate University for Advanced Studies)（综合研究大学院大学）；National Institute of Informatics（国立信息学研究所）
Detecting salient objects from a video requires exploiting both spatial and temporal knowledge included in the video. We propose a novel region-based multiscale spatiotemporal saliency detection method for videos, where static features and dynamic features computed from the low and middle levels are combined together. Our method utilizes such combined features spatially over each frame and, at the same time, temporally across frames using consistency between consecutive frames. Saliency cues in our method are analyzed through a multiscale segmentation model, and fused across scale levels, yiel…
Apple Inc.（苹果公司）；Mercer University（莫瑟尔大学）；Schlumberger Ltd.（斯伦贝谢有限公司）；Georgia Institute of Technology（佐治亚理工学院）
A new approach to seismic interpretation is proposed to leverage visual perception and human visual system modeling. Specifically, a saliency detection algorithm based on a novel attention model is proposed for identifying subsurface structures within seismic data volumes. The algorithm employs 3D-FFT and a multi-dimensional spectral projection, which decomposes local spectra into three distinct components, each depicting variations along different dimensions of the data. Subsequently, a novel directional center-surround attention model is proposed to incorporate directional comparisons around…
Nanyang Technological University（南洋理工大学）；Beijing Jiaotong University（北京交通大学）；Dalian University of Technology（大连理工大学）；Chinese Academy of Sciences（中国科学院）
We present an effective method to progressively integrate and refine the cross-modality complementarities for RGB-D salient object detection (SOD). The proposed network mainly solves two challenging issues: 1) how to effectively integrate the complementary information from RGB image and its corresponding depth map, and 2) how to adaptively select more saliency-related features. First, we propose a cross-modality feature modulation (cmFM) module to enhance feature representations by taking the depth features as prior, which models the complementary relations of RGB-D data. Second, we propose an…
Spectrum-driven Mixed-frequency Network for Hyperspectral Salient Object Detection
Beijing Institute of Technology（北京理工大学）；Key Laboratory of Photoelectronic Imaging Technology and System, Ministry of Education of China（教育部光电成像技术与系统重点实验室）；Chongqing Innovation Center（重庆创新中心）
We propose a novel iterative method to adapt a a graph to d-dimensional image data. The method drives the nodes of the graph towards image features. The adaptation process naturally lends itself to a measure of feature saliency which can then be used to retain meaningful nodes and edges in the graph. From the adapted graph, we also propose the computation of a dual graph, which inherits the saliency measure from the adapted graph, and whose edges run along image features, hence producing an oversegmenting graph. The proposed method is computationally efficient and fully parallelisable. We prop…
Recent frameworks, such as the so-called plug-and-play, allow us to leverage the developments in image denoising to tackle other, and more involved, problems in image processing. As the name suggests, state-of-the-art denoisers are plugged into an iterative algorithm that alternates between a denoising step and the inversion of the observation operator. While these tools offer flexibility, the convergence of the resulting algorithm may be difficult to analyse. In this paper, we plug a state-of-the-art denoiser, based on a Gaussian mixture model, in the iterations of an alternating direction me…
Tongji University（同济大学）；the University of Hong Kong（香港大学）
Vision-based object tracking has boosted extensive autonomous applications for unmanned aerial vehicles (UAVs). However, the dynamic changes in flight maneuver and viewpoint encountered in UAV tracking pose significant difficulties, e.g. , aspect ratio change, and scale variation. The conventional cross-correlation operation, while commonly used, has limitations in effectively capturing perceptual similarity and incorporates extraneous background information. To mitigate these limitations, this work presents a novel saliency-guided dynamic vision Transformer (SGDViT) for UAV tracking. The prop…
Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）
In this paper, we propose an end-to-end group-wise deep co-saliency detection approach to address the co-salient object discovery problem based on the fully convolutional network (FCN) with group input and group output. The proposed approach captures the group-wise interaction information for group images by learning a semantics-aware image representation based on a convolutional neural network, which adaptively learns the group-wise features for co-saliency detection. Furthermore, the proposed approach discovers the collaborative and interactive relationships between group-wise feature repres…
University of Dayton（代顿大学）；National University of Singapore（新加坡国立大学）
Salient object detection has increasingly become a popular topic in cognitive and computational sciences, including computer vision and artificial intelligence research. In this paper, we propose integrating \textit{semantic priors} into the salient object detection process. Our algorithm consists of three basic steps. Firstly, the explicit saliency map is obtained based on the semantic segmentation refined by the explicit saliency priors learned from the data. Next, the implicit saliency map is computed based on a trained model which maps the implicit saliency priors embedded into regional fe…
Instituto de Telecomunicações（电信研究所）；Instituto Superior Técnico（高等技术学院）
The recently proposed plug-and-play (PnP) framework allows leveraging recent developments in image denoising to tackle other, more involved, imaging inverse problems. In a PnP method, a black-box denoiser is plugged into an iterative algorithm, taking the place of a formal denoising step that corresponds to the proximity operator of some convex regularizer. While this approach offers flexibility and excellent performance, convergence of the resulting algorithm may be hard to analyze, as most state-of-the-art denoisers lack an explicit underlying objective function. In this paper, we propose a …
University of Alberta（阿尔伯塔大学）；Wuhan University（武汉大学）；Dalian University of Technology（大连理工大学）
Growing interests in RGB-D salient object detection (RGB-D SOD) have been witnessed in recent years, owing partly to the popularity of depth sensors and the rapid progress of deep learning techniques. Unfortunately, existing RGB-D SOD methods typically demand large quantity of training images being thoroughly annotated at pixel-level. The laborious and time-consuming manual annotation has become a real bottleneck in various practical scenarios. On the other hand, current unsupervised RGB-D SOD methods still heavily rely on handcrafted feature representations. This inspires us to propose in thi…
Utah State University（犹他州立大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Idaho（爱达荷大学）；The Affiliated Hospital of Qingdao University（青岛大学附属医院）；The Second Hospital of Hebei Medical University（河北医科大学第二医院）
Tumor saliency estimation aims to localize tumors by modeling the visual stimuli in medical images. However, it is a challenging task for breast ultrasound due to the complicated anatomic structure of the breast and poor image quality; and existing saliency estimation approaches only model generic visual stimuli, e.g., local and global contrast, location, and feature correlation, and achieve poor performance for tumor saliency estimation. In this paper, we propose a novel optimization model to estimate tumor saliency by utilizing breast anatomy. First, we model breast anatomy and decompose bre…
Existing deep learning-based Unsupervised Salient Object Detection (USOD) methods rely on supervised pre-trained deep models. Moreover, they generate pseudo labels based on hand-crafted features, which lack high-level semantic information. In order to overcome these shortcomings, we propose a new two-stage Activation-to-Saliency (A2S) framework that effectively excavates high-quality saliency cues to train a robust saliency detector. It is worth noting that our method does not require any manual annotation, even in the pre-training phase. In the first stage, we transform an unsupervisedly pre-…
Despite significant progress, image saliency detection still remains a challenging task in complex scenes and environments. Integrating multiple different but complementary cues, like RGB and Thermal (RGB-T), may be an effective way for boosting saliency detection performance. The current research in this direction, however, is limited by the lack of a comprehensive benchmark. This work contributes such a RGB-T image dataset, which includes 821 spatially aligned RGB-T image pairs and their ground truth annotations for saliency detection purpose. The image pairs are with high diversity recorded…
Recent advances in deep learning have markedly improved the quality of visual-attention modelling. In this work we apply these advances to video compression. We propose a compression method that uses a saliency model to adaptively compress frame areas in accordance with their predicted saliency. We selected three state-of-the-art saliency models, adapted them for video compression and analyzed their results. The analysis includes objective evaluation of the models as well as objective and subjective evaluation of the compressed videos. Our method, which is based on the x264 video codec, can pr…
