Phrase detection requires methods to identify if a phrase is relevant to an image and localize it, if applicable. A key challenge for training more discriminative detection models is sampling negatives. Sampling techniques from prior work focus primarily on hard, often noisy, negatives disregarding the broader distribution of negative samples. Our proposed CFCD-Net addresses this through two novels methods. First, we generate groups of semantically similar words we call concepts (\eg, \{dog, cat, horse\} and \ \{car, truck, SUV\}), and then train our CFCD-Net to discriminate between a region of interest and its unrelated concepts. Second, for phrases containing fine-grained mutually-exclusive words (\eg, colors), we force the model to select only one applicable phrase for each region using our novel fine-grained module (FGM). We evaluate our approach on Flickr30K Entities and RefCOCO+, where we improve mAP over the state-of-the-art by 1.5-2 points. When considering only the phrases affected by our FGM module, we improve by 3-4 points on both datasets.
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；University of Science and Technology of China（中国科学技术大学）；Xiaomi AI Lab（小米人工智能实验室）
Image-text matching has received growing interest since it bridges vision and language. The key challenge lies in how to learn correspondence between image and text. Existing works learn coarse correspondence based on object co-occurrence statistics, while failing to learn fine-grained phrase correspondence. In this paper, we present a novel Graph Structured Matching Network (GSMN) to learn fine-grained correspondence. The GSMN explicitly models object, relation and attribute as a structured phrase, which not only allows to learn correspondence of object, relation and attribute separately, but…
Microsoft（微软）；University of California Los Angeles（加利福尼亚大学洛杉矶分校）；New York University（纽约大学）
Vision-language (VL) pre-training has recently received considerable attention. However, most existing end-to-end pre-training approaches either only aim to tackle VL tasks such as image-text retrieval, visual question answering (VQA) and image captioning that test high-level understanding of images, or only target region-level understanding for tasks such as phrase grounding and object detection. We present FIBER (Fusion-In-the-Backbone-based transformER), a new VL model architecture that can seamlessly handle both these types of tasks. Instead of having dedicated transformer layers for fusio…
Dynamic Prompting of Frozen Text-to-Image Diffusion Models for Panoptic Narrative Grounding
School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；School of Software, Beihang University（北京航空航天大学软件学院）；Meituan（美团）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程学院）
GUIDED: Granular Understanding via Identification, Detection, and Discrimination for Fine-Grained Open-Vocabulary Object Detection
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Meituan（美团）；GuangDong Province Key Laboratory of Information Security Technology（广东省信息安全技术重点实验室）；Research Institute, Sun Yat-sen University（中山大学深圳研究院）
Few-shot learning aims at rapidly adapting to novel categories with only a handful of samples at test time, which has been predominantly tackled with the idea of meta-learning. However, meta-learning approaches essentially learn across a variety of few-shot tasks and thus still require large-scale training data with fine-grained supervision to derive a generalized model, thereby involving prohibitive annotation cost. In this paper, we advance the few-shot classification paradigm towards a more challenging scenario, i.e., cross-granularity few-shot classification, where the model observes only …
Sea AI Lab（Sea人工智能实验室）；National University of Singapore（新加坡国立大学）
In self-supervised learning, multi-granular features are heavily desired though rarely investigated, as different downstream tasks (e.g., general and fine-grained classification) often require different or multi-granular features, e.g.~fine- or coarse-grained one or their mixture. In this work, for the first time, we propose an effective MUlti-Granular Self-supervised learning (Mugs) framework to explicitly learn multi-granular visual features. Mugs has three complementary granular supervisions: 1) an instance discrimination supervision (IDS), 2) a novel local-group discrimination supervision …
Aselsan Research Center（阿瑟尔桑研究中心）；EPFL（洛桑联邦理工学院）；Center for Image Analysis (OGAM)（图像分析中心（OGAM））；METU（中东技术大学）
Recognition of objects with subtle differences has been used in many practical applications, such as car model recognition and maritime vessel identification. For discrimination of the objects in fine-grained detail, we focus on deep embedding learning by using a multi-task learning framework, in which the hierarchical labels (coarse and fine labels) of the samples are utilized both for classification and a quadruplet-based loss function. In order to improve the recognition strength of the learned features, we present a novel feature selection method specifically designed for four training sam…
Recent efforts of multimodal Transformers have improved Visually Rich Document Understanding (VrDU) tasks via incorporating visual and textual information. However, existing approaches mainly focus on fine-grained elements such as words and document image patches, making it hard for them to learn from coarse-grained elements, including natural lexical units like phrases and salient visual regions like prominent image regions. In this paper, we attach more importance to coarse-grained elements containing high-density information and consistent semantics, which are valuable for document understa…
School of Computer Science and Software Engineering, Southwest Petroleum University（计算机科学与软件工程学院，西南石油大学）；School of Computing and Artificial Intelligence, Southwest Jiaotong University（计算与人工智能学院，西南交通大学）；School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics（计算与人工智能学院，西南财经大学）
School of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Instrumentation Science and Engineering, Harbin Institute of Technology（哈尔滨工业大学仪器科学与工程学院）；Harbin Institute of Technology Suzhou Research Institute（哈尔滨工业大学苏州研究院）；Ningbo Haichuang Group Co., Ltd.（宁波海创集团有限公司）；School of Software Technology, Zhejiang University（浙江大学软件学院）；School of Information Engineering, North China University of Water Resources and Electric Power（中国北方水利电力大学信息工程学院）；Department of Geo-spatial Information, Information Engineering University（信息工程大学地理信息学院）；Innovation Center of Yangtze River Delta, Zhejiang University（长三角创新中心，浙江大学）
Extracting discriminative features plays a crucial role in the fine-grained visual classification task. Most of the existing methods focus on developing attention or augmentation mechanisms to achieve this goal. However, addressing the ambiguity in the top-k prediction classes is not fully investigated. In this paper, we introduce a Self Assessment Classifier, which simultaneously leverages the representation of the image and top-k prediction classes to reassess the classification results. Our method is inspired by continual learning with coarse-grained and fine-grained classifiers to increase…
Anatomy-Aware Text-Visual Fusion with Dual-Perspective Prompts for Fine-Grained Lumbar Spine Segmentation
College of Computer and Data Science（计算机与数据科学学院）；Shandong Technology and Business University（山东科技商务大学）；Engineering Research Center of Big Data Intelligence (Fuzhou University)（大数据智能工程研究中心（福州大学））；The Chinese University of Hong Kong（香港中文大学）；Case Western Reserve University（凯斯西储大学）
University of Cambridge（剑桥大学）；Huawei R&D UK（华为英国研发中心）
Multi-Modal Self-Supervised Learning from videos has been shown to improve model's performance on various downstream tasks. However, such Self-Supervised pre-training requires large batch sizes and a large amount of computation resources due to the noise present in the uncurated data. This is partly due to the fact that the prevalent training scheme is trained on coarse-grained setting, in which vectors representing the whole video clips or natural language sentences are used for computing similarity. Such scheme makes training noisy as part of the video clips can be totally not correlated wit…
From Classification to Cross-Modal Understanding: Leveraging Vision-Language Models for Fine-Grained Renal Pathology
New York University（纽约大学）；Cornell Tech（康奈尔科技）；Vanderbilt University（范德比大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Regensburg（莱茵河畔大学）；Weill Cornell Medicine（韦尔·科恩医学中心）；Northwell Health（北well健康）
Southwest Jiaotong University（西南交通大学）；National University of Singapore（新加坡国立大学）
Fine-grained object classification is a challenging task due to the subtle inter-class difference and large intra-class variation. Recently, visual attention models have been applied to automatically localize the discriminative regions of an image for better capturing critical difference and demonstrated promising performance. However, without consideration of the diversity in attention process, most of existing attention models perform poorly in classifying fine-grained objects. In this paper, we propose a diversified visual attention network (DVAN) to address the problems of fine-grained obj…
Proto-FG3D: Prototype-based Interpretable Fine-Grained 3D Shape Classification
School of Information Science and Engineering（信息科学与工程学院）；University of Jinan（济南大学）；School of Control Science and Engineering（控制科学与工程学院）；Shandong University（山东大学）；Department of Computing and Decision Sciences（计算与决策科学系）；Lingnan University（岭南大学）；College of Computer Science（计算机科学学院）；Nankai University（南开大学）
From Region to Patch: Attribute-Aware Foreground-Background Contrastive Learning for Fine-Grained Fashion Retrieval
Zhejiang Gongshang University（浙江工商大学）；Zhejiang University（浙江大学）；Peking University（北京大学）；Huazhong University of Science and Technology（华中科技大学）；University of Science and Technology of China（中国科学技术大学）
