Most of the saliency methods are evaluated on their ability to generate saliency maps, and not on their functionality in a complete vision pipeline, like for instance, image classification. In the current paper, we propose an approach which does not require explicit saliency maps to improve image classification, but they are learned implicitely, during the training of an end-to-end image classification task. We show that our approach obtains similar results as the case when the saliency maps are provided explicitely. Combining RGB data with saliency maps represents a significant advantage for object recognition, especially for the case when training data is limited. We validate our method on several datasets for fine-grained classification tasks (Flowers, Birds and Cars). In addition, we show that our saliency estimation method, which is trained without any saliency groundtruth data, obtains competitive results on real image saliency benchmark (Toronto), and outperforms deep saliency models with synthetic images (SID4VAM).
Computer Vision Center（计算机视觉中心）；Universidad del Bío Bío（比奥比奥大学）
This paper investigates the role of saliency to improve the classification accuracy of a Convolutional Neural Network (CNN) for the case when scarce training data is available. Our approach consists in adding a saliency branch to an existing CNN architecture which is used to modulate the standard bottom-up visual features from the original image input, acting as an attentional mechanism that guides the feature extraction process. The main aim of the proposed approach is to enable the effective training of a fine-grained recognition model with limited training samples and to improve the perform…
Australian National University（澳大利亚国立大学）；Data61/CSIRO（Data61/联邦科学与工业研究组织）
Learning new concepts from a few of samples is a standard challenge in computer vision. The main directions to improve the learning ability of few-shot training models include (i) a robust similarity learning and (ii) generating or hallucinating additional data from the limited existing samples. In this paper, we follow the latter direction and present a novel data hallucination model. Currently, most datapoint generators contain a specialized network (i.e., GAN) tasked with hallucinating new datapoints, thus requiring large numbers of annotated data for their training in the first place. In t…
The Chinese University of Hong Kong1,Geng Mei AI lab[2, 4], The University of Hong Kong3.（香港中文大学1，羹美人工智能实验室[2, 4]，香港大学3.）
Fine-grained image classification is a challenging problem, since the difficulty of finding discriminative features. To handle this circumstance, basically, there are two ways to go. One is use attention based method to focus on informative areas, while the other one aims to find high order between features. Further, for attention based method there are two directions, activation based and detection based, which are proved effective by scholars. However ,rare work focus on fusing two types of attention with high order feature. In this paper, we propose a novel DAF method which fuse two types o…
Self-supervised contrastive learning has demonstrated great potential in learning visual representations. Despite their success in various downstream tasks such as image classification and object detection, self-supervised pre-training for fine-grained scenarios is not fully explored. We point out that current contrastive methods are prone to memorizing background/foreground texture and therefore have a limitation in localizing the foreground object. Analysis suggests that learning to extract discriminative texture information and localization are equally crucial for fine-grained self-supervis…
University of Catania（卡塔尼亚大学）；Simula Research Lab（Simula研究实验室）；University of Oslo（奥斯陆大学）
This paper presents an approach for top-down saliency detection guided by visual classification tasks. We first learn how to compute visual saliency when a specific visual task has to be accomplished, as opposed to most state-of-the-art methods which assess saliency merely through bottom-up principles. Afterwards, we investigate if and to what extent visual saliency can support visual classification in nontrivial cases. To achieve this, we propose SalClassNet, a CNN framework consisting of two networks jointly trained: a) the first one computing top-down saliency maps from input images, and b)…
AI4Health Division, Faculty VI Carl von Ossietzky Universität Oldenburg（AI4Health部门，第六学院卡尔·冯·奥西特齐克大学）；Semantic Information Systems Group Osnabrück University（语义信息系统小组奥尔纳布里克大学）
Robust Saliency-Aware Distillation for Few-shot Fine-grained Visual Recognition
School of Computer Science and Engineering, South China University of Technology（华南理工大学计算机科学与工程学院）；Brain and Affective Cognitive Research Center, Pazhou Lab（琶洲实验室脑与情感认知研究中心）；Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human（教育部健康智能感知与平行数字人工程研究中心）
Harbin Institute of Technology（哈尔滨工业大学）；SenseTime Research（商汤科技研究院）；Hong Kong Polytechnic University（香港理工大学）；Damo Academy, Alibaba Group（阿里巴巴集团达摩院）
In recent years, hashing methods have been proved to be effective and efficient for the large-scale Web media search. However, the existing general hashing methods have limited discriminative power for describing fine-grained objects that share similar overall appearance but have subtle difference. To solve this problem, we for the first time introduce the attention mechanism to the learning of fine-grained hashing codes. Specifically, we propose a novel deep hashing model, named deep saliency hashing (DSaH), which automatically mines salient regions and learns semantic-preserving hashing code…
FGSGT: Saliency-Guided Siamese Network Tracker Based on Key Fine-Grained Feature Information for Thermal Infrared Target Tracking
College of Computer and Information Technology, China Three Gorges University（三峡大学计算机与信息学院）；Hubei Province Engineering Technology Research Center for Construction Quality Testing Equipment, China Three Gorges University（三峡大学湖北省建筑质量检测装备工程技术研究中心）；Nantong Institute of Technology（南通理工学院）；Wuhan College（武汉学院）
LGEST: Dynamic Spatial-Spectral Expert Routing for Hyperspectral Image Classification
Information Hub, The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学信息中心（广州））；Faculty of Geographical Science, Beijing Normal University（北京师范大学地理学部）；School of Electronic and Communication Engineering, Guangzhou University（广州大学电子与通信工程学院）
Discriminative localization is essential for fine-grained image classification task, which devotes to recognizing hundreds of subcategories in the same basic-level category. Reflecting on discriminative regions of objects, key differences among different subcategories are subtle and local. Existing methods generally adopt a two-stage learning framework: The first stage is to localize the discriminative regions of objects, and the second is to encode the discriminative features for training classifiers. However, these methods generally have two limitations: (1) Separation of the two-stage learn…
Salient object detection increasingly receives attention as an important component or step in several pattern recognition and image processing tasks. Although a variety of powerful saliency models have been intensively proposed, they usually involve heavy feature (or model) engineering based on priors (or assumptions) about the properties of objects and backgrounds. Inspired by the effectiveness of recently developed feature learning, we provide a novel Deep Image Saliency Computing (DISC) framework for fine-grained image saliency computing. In particular, we model the image saliency from both…
Tianjin University（天津大学）；University of Warwick（华威大学）
This paper studies the task of matching image and sentence, where learning appropriate representations across the multi-modal data appears to be the main challenge. Unlike previous approaches that predominantly deploy symmetrical architecture to represent both modalities, we propose Saliency-guided Attention Network (SAN) that asymmetrically employs visual and textual attention modules to learn the fine-grained correlation intertwined between vision and language. The proposed SAN mainly includes three components: saliency detector, Saliency-weighted Visual Attention (SVA) module, and Saliency-…
SGD-Mix: Enhancing Domain-Specific Image Classification with Label-Preserving Data Augmentation
Department of Computer Science and Information Engineering, National Cheng Kung University（计算机科学与信息工程系，国立成功大学）；Department of Biomedical Informatics, Harvard Medical School, Harvard University（生物医学信息学系，哈佛医学院，哈佛大学）
Boundary Attention Mapping (BAM): Fine-grained saliency maps for segmentation of Burn Injuries
Skinopathy Inc.（Skinopathy公司）；Carleton University（卡尔顿大学）；University of Alberta Hospital（阿尔伯塔大学医院）；Scarborough Health Network, Centenary Hospital（士嘉堡健康网络百年纪念医院）；University of Montreal（蒙特利尔大学）；Quebec AI Institute (MILA)（魁北克人工智能研究所（MILA））
Sun Yat-sen University（中山大学）；Advanced Digital Sciences Center（先进数字科学中心）；Anhui University（安徽大学）
Driven by recent vision and graphics applications such as image segmentation and object recognition, computing pixel-accurate saliency values to uniformly highlight foreground objects becomes increasingly important. In this paper, we propose a unified framework called PISA, which stands for Pixelwise Image Saliency Aggregating various bottom-up cues and priors. It generates spatially coherent yet detail-preserving, pixel-accurate and fine-grained saliency, and overcomes the limitations of previous methods which use homogeneous superpixel-based and color only treatment. PISA aggregates multiple…
Image Landmark Recognition has been one of the most sought-after classification challenges in the field of vision and perception. After so many years of generic classification of buildings and monuments from images, people are now focussing upon fine-grained problems - recognizing the category of each building or monument. We proposed an ensemble network for the purpose of classification of Indian Landmark Images. To this end, our method gives robust classification by ensembling the predictions from Graph-Based Visual Saliency (GBVS) network alongwith supervised feature-based classification al…
Sentiment analysis is crucial for extracting social signals from social media content. Due to the prevalence of images in social media, image sentiment analysis is receiving increasing attention in recent years. However, most existing systems are black-boxes that do not provide insight on how image content invokes sentiment and emotion in the viewers. Psychological studies have confirmed that salient objects in an image often invoke emotions. In this work, we investigate more fine-grained and more comprehensive interaction between visual saliency and visual sentiment. In particular, we partiti…
Massachusetts Institute of Technology（麻省理工学院）；Toyota Research Institute（丰田研究院）
We introduce a saliency-based distortion layer for convolutional neural networks that helps to improve the spatial sampling of input data for a given task. Our differentiable layer can be added as a preprocessing block to existing task networks and trained altogether in an end-to-end fashion. The effect of the layer is to efficiently estimate how to sample from the original data in order to boost task performance. For example, for an image classification task in which the original data might range in size up to several megapixels, but where the desired input images to the task network are much…
Northwestern Polytechnical University（西北工业大学）；Inception Institute of Artificial Intelligence（人工智能初始研究所）
Humans perform co-saliency detection by first summarizing the consensus knowledge in the whole group and then searching corresponding objects in each image. Previous methods usually lack robustness, scalability, or stability for the first process and simply fuse consensus features with image features for the second process. In this paper, we propose a novel consensus-aware dynamic convolution model to explicitly and effectively perform the "summarize and search" process. To summarize consensus image features, we first summarize robust features for every single image using an effective pooling …
DecomCAM: Advancing Beyond Saliency Maps through Decomposition and Integration
Beihang University（北京航空航天大学）；Zhongguancun Laboratory（中关村实验室）；Academy of Military Science Defense Innovation Institute（军事科学院国防创新研究院）；Communication University of China（中国传媒大学）
Learning Fine-grained Domain Generalization via Hyperbolic State Space Hallucination
School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；Faculty of Information Technology, Monash University（莫纳什大学信息技术学院）；School of Medicine, Yale University（耶鲁大学医学院）
Sejong University（世宗大学）；Abdul Wali Khan University（阿卜杜勒·瓦利·汗大学）；Australian National University（澳大利亚国立大学）；Data61-CSIRO（澳大利亚联邦科学与工业研究组织数据61研究所）；University of Technology Sydney（悉尼科技大学）；University of Canberra（堪培拉大学）
Salient object detection (SOD) extracts meaningful contents from an input image. RGB-based SOD methods lack the complementary depth clues; hence, providing limited performance for complex scenarios. Similarly, RGB-D models process RGB and depth inputs, but the depth data availability during testing may hinder the model's practical applicability. This paper exploits only RGB images, estimates depth from RGB, and leverages the intermediate depth features. We employ a pyramidal attention structure to extract multi-level convolutional-transformer features to process initial stage representations a…
University of Science and Technology of China（中国科学技术大学）；Southeast University（东南大学）；Alibaba Group（阿里巴巴集团）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Current metrics for video captioning are mostly based on the text-level comparison between reference and candidate captions. However, they have some insuperable drawbacks, e.g., they cannot handle videos without references, and they may result in biased evaluation due to the one-to-many nature of video-to-text and the neglect of visual relevance. From the human evaluator's viewpoint, a high-quality caption should be consistent with the provided video, but not necessarily be similar to the reference in literal or semantics. Inspired by human evaluation, we propose EMScore (Embedding Matching-ba…
High-resolution images are widely adopted for high-performance object detection in videos. However, processing high-resolution inputs comes with high computation costs, and naive down-sampling of the input to reduce the computation costs quickly degrades the detection performance. In this paper, we propose SALISA, a novel non-uniform SALiency-based Input SAmpling technique for video object detection that allows for heavy down-sampling of unimportant background regions while preserving the fine-grained details of a high-resolution image. The resulting image is spatially smaller, leading to redu…
Cross-attention-based saliency inference for predicting cancer metastasis on whole slide images
Center for Biomedical Informatics, Wake Forest University School of Medicine（维克森林大学医学院生物医学信息学中心）；Department of Pathology, Wake Forest University School of Medicine（维克森林大学医学院病理学系）
University of Florida（佛罗里达大学）；University of Minnesota, Twin Cities（明尼苏达大学双城分校）
This paper presents a holistic approach to saliency-guided visual attention modeling (SVAM) for use by autonomous underwater robots. Our proposed model, named SVAM-Net, integrates deep visual features at various scales and semantics for effective salient object detection (SOD) in natural underwater images. The SVAM-Net architecture is configured in a unique way to jointly accommodate bottom-up and top-down learning within two separate branches of the network while sharing the same encoding layers. We design dedicated spatial attention modules (SAMs) along these learning pathways to exploit the…
Saliency methods generating visual explanatory maps representing the importance of image pixels for model classification is a popular technique for explaining neural network decisions. Hierarchical dynamic masks (HDM), a novel explanatory maps generation method, is proposed in this paper to enhance the granularity and comprehensiveness of saliency maps. First, we suggest the dynamic masks (DM), which enables multiple small-sized benchmark mask vectors to roughly learn the critical information in the image through an optimization method. Then the benchmark mask vectors guide the learning of lar…
Zhejiang University（浙江大学）；University of California, Santa Cruz（加利福尼亚大学圣克鲁兹分校）；University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校）
Grounded video description (GVD) encourages captioning models to attend to appropriate video regions (e.g., objects) dynamically and generate a description. Such a setting can help explain the decisions of captioning models and prevents the model from hallucinating object words in its description. However, such design mainly focuses on object word generation and thus may ignore fine-grained information and suffer from missing visual concepts. Moreover, relational words (e.g., "jump left or right") are usual spatio-temporal inference results, i.e., these words cannot be grounded on certain spat…
From Attenuation to Attention: Variational Information Flow Manipulation for Fine-Grained Visual Perception
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)（智能信息处理重点实验室，计算技术研究所，中国科学院（ICT/CAS））；State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)（人工智能安全国家重点实验室，计算技术研究所，中国科学院（ICT/CAS））；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京，中国）
NEC Labs America（美国NEC实验室）；University of Iowa（爱荷华大学）；University of Missouri（密苏里大学）
This paper proposes to go beyond the state-of-the-art deep convolutional neural network (CNN) by incorporating the information from object detection, focusing on dealing with fine-grained image classification. Unfortunately, CNN suffers from over-fiting when it is trained on existing fine-grained image classification benchmarks, which typically only consist of less than a few tens of thousands training images. Therefore, we first construct a large-scale fine-grained car recognition dataset that consists of 333 car classes with more than 150 thousand training images. With this large-scale datas…
PestVL-Net: Enabling Multimodal Pest Learning via Fine-grained Vision-Language Interaction
Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences（智能机器研究所，合肥物理科学研究院，中国科学院）；University of Science and Technology of China（中国科学技术大学）；Zhongke Hefei Institute of Technology Innovation Engineering（中科合肥技术创新工程研究院）
University of Oxford（牛津大学）；University of Edinburgh（爱丁堡大学）
The different families of saliency methods, either based on contrastive signals, closed-form formulas mixing gradients with activations or on perturbation masks, all focus on which parts of an image are responsible for the model's inference. In this paper, we are rather interested by the locations of an image that contribute to the model's training. First, we propose a principled attribution method that we extract from the summation formula used to compute the gradient of the weights for a 1x1 convolutional layer. The resulting formula is fast to compute and can used throughout the network, al…
Discovering Fine-Grained Visual-Concept Relations by Disentangled Optimal Transport Concept Bottleneck Models
National Key Laboratory of Radar Signal Processing, Xidian University, Xi’an, 710071, China（雷达信号处理国家级重点实验室，西安电子科技大学）；State Key Laboratory of Integrated Service Networks, Xidian University, Xi’an, 710071, China（集成服务网络国家重点实验室，西安电子科技大学）
LICO: Explainable Models with Language-Image Consistency
Fudan University（复旦大学）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；Shanghai Center for Brain Science and Brain-inspired Technology（上海脑科学与类脑研究中心）
Fine-grained Background Representation for Weakly Supervised Semantic Segmentation
Korea Advanced Institute of Science and Technology（韩国科学技术院）；Ewha Womans University（梨花女子大学）；Zhejiang University（浙江大学）；Zhejiang Lab（之江实验室）；University of Michigan（密歇根大学）
Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysis
The Ohio State University（俄亥俄州立大学）；University of Tsukuba（筑波大学）；Virginia Tech（弗吉尼亚理工大学）；Princeton University（普林斯顿大学）；Rensselaer Polytechnic Institute（伦斯勒理工学院）
Histopathological images provide rich information for disease diagnosis. Large numbers of histopathological images have been digitized into high resolution whole slide images, opening opportunities in developing computational image analysis tools to reduce pathologists' workload and potentially improve inter- and intra- observer agreement. Most previous work on whole slide image analysis has focused on classification or segmentation of small pre-selected regions-of-interest, which requires fine-grained annotation and is non-trivial to extend for large-scale whole slide analysis. In this paper,…
In this paper, we present a framework that jointly retrieves and spatiotemporally highlights actions in videos by enhancing current deep cross-modal retrieval methods. Our work takes on the novel task of action highlighting, which visualizes where and when actions occur in an untrimmed video setting. Action highlighting is a fine-grained task, compared to conventional action recognition tasks which focus on classification or window-based localization. Leveraging weak supervision from annotated captions, our framework acquires spatiotemporal relevance maps and generates local embeddings which r…
