Focus, Distinguish, and Prompt: Unleashing CLIP for Efficient and Flexible Scene Text Retrieval
School of Cyber Science and Engineering, Nanjing University of Science and Technology（南京理工大学网络空间安全学院）；State Key Laboratory of Media Convergence and Communication, Communication University of China（中国传媒大学媒体融合与传播国家重点实验室）；Lenovo Research（联想研究院）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Laboratory for Advanced Computing and Intelligence Engineering（先进计算与智能工程实验室）；College of Computer Science, Nankai University（南开大学计算机学院）
场景文本检索旨在从图像库中找出所有包含查询文本的图像。当前方法倾向于采用光学字符识别（OCR）流程，这需要复杂的文本检测和/或识别过程，导致检索效率低且灵活性差。与这些方法不同，本文提出探索对比语言-图像预训练（CLIP）在无OCR场景文本检索中的内在潜力。通过实证分析，我们观察到CLIP作为文本检索器的主要挑战是：1）有限的文本感知尺度，以及2）纠缠的视觉-语义概念。为此，我们提出了一种名为FDP（Focus, Distinguish, and Prompt）的新模型。FDP首先通过将注意力转移到文本区域并探测隐藏的文本知识来聚焦场景文本，然后将查询文本分为实词和功能词进行处理，其中采用了语义感知提示方案和干扰查询辅助模块。大量实验表明，与现有方法相比，FDP显著提高了推理速度，同时取得了更好或具有竞争力的检索精度。值得注意的是，在IIIT-STR基准上，FDP以4倍的速度超越最先进模型4.37%。此外，在短语级和属性感知场景文本检索设置下的额外实验验证了FDP在处理多种形式查询文本方面的独特优势。源代码将在https://github.com/Gyann-z/FDP公开。
英文摘要：
Scene text retrieval aims to find all images containing the query text from an image gallery. Current efforts tend to adopt an Optical Character Recognition (OCR) pipeline, which requires complicated text detection and/or recognition processes, resulting in inefficient and inflexible retrieval. Different from them, in this work we propose to explore the intrinsic potential of Contrastive Language-Image Pre-training (CLIP) for OCR-free scene text retrieval. Through empirical analysis, we observe that the main challenges of CLIP as a text retriever are: 1) limited text perceptual scale, and 2) entangled visual-semantic concepts. To this end, a novel model termed FDP (Focus, Distinguish, and Prompt) is developed. FDP first focuses on scene text via shifting the attention to the text area and probing the hidden text knowledge, and then divides the query text into content word and function word for processing, in which a semantic-aware prompting scheme and a distracted queries assistance module are utilized. Extensive experiments show that FDP significantly enhances the inference speed while achieving better or competitive retrieval accuracy compared to existing methods. Notably, on the IIIT-STR benchmark, FDP surpasses the state-of-the-art model by 4.37% with a 4 times faster speed. Furthermore, additional experiments under phrase-level and attribute-aware scene text retrieval settings validate FDP's particular advantages in handling diverse forms of query text. The source code will be publicly available at https://github.com/Gyann-z/FDP.
Unleashing the Potential of All Test Samples: Mean-Shift Guided Test-Time Adaptation
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（人机混合增强智能国家重点实验室，人工智能与机器人研究院，西安交通大学）；Faculty of Computility Microelectronics, Shenzhen University of Advanced Technology and the Guangdong Provincial Key Laboratory of Computility Microelectronics（计算微电子学系，深圳先进技术大学及广东省计算微电子重点实验室）；College of Software Engineering, Xi’an Jiaotong University（软件工程学院，西安交通大学）
Unleashing the Power of Vision-Language Models for Long-Tailed Multi-Label Visual Recognition
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of Computer Network and Information Integration (Southeast University)（计算机网络与信息集成重点实验室）；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（Mohamed bin Zayed人工智能大学）；Carnegie Mellon University（卡内基梅隆大学）
Unleashing Diffusion and State Space Models for Medical Image Segmentation
Department of Biostatistics, School of Global Public Health, New York University（生物统计学系、全球公共卫生学院、纽约大学）；School of Statistics, KLATASDS-MOE, East China Normal University（统计学系、KLATASDS-MOE、华东师范大学）；School of Biomedical Engineering, Southern Medical University（生物医学工程学院、南方医科大学）；Faculty of Biomedical Engineering, Shenzhen University of Advanced Technology（生物医学工程学院、深圳先进技术大学）
UC Santa Cruz（加州大学圣克鲁兹分校）；Johns Hopkins University（约翰斯·霍普金斯大学）；Shanghai Jiao Tong University（上海交通大学）
This paper presents a simple and effective visual prompting method for adapting pre-trained models to downstream recognition tasks. Our method includes two key designs. First, rather than directly adding together the prompt and the image, we treat the prompt as an extra and independent learnable component. We show that the strategy of reconciling the prompt and the image matters, and find that warping the prompt around a properly shrinked image empirically works the best. Second, we re-introduce two "old tricks" commonly used in building transferable adversarial examples, i.e., input diversity…
Unleashing the Power of Prompt-driven Nucleus Instance Segmentation
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Engineering, Westlake University（西湖大学工学院）；University of Liverpool（利物浦大学）
Modern image captioning models are usually trained with text similarity objectives. However, since reference captions in public datasets often describe the most salient common objects, models trained with text similarity objectives tend to ignore specific and detailed aspects of an image that distinguish it from others. Toward more descriptive and distinctive caption generation, we propose using CLIP, a multimodal encoder trained on huge image-text pairs from web, to calculate multimodal similarity and use it as a reward function. We also propose a simple finetuning strategy of the CLIP text e…
McGill University（麦吉尔大学）；City University of Hong Kong（香港城市大学）；South China University of Technology（华南理工大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
Image captioning models are usually trained according to human annotated ground-truth captions, which could generate accurate but generic captions. In this paper, we focus on generating distinctive captions that can distinguish the target image from other similar images. To evaluate the distinctiveness of captions, we introduce a series of metrics that use large-scale vision-language pre-training model CLIP to quantify the distinctiveness. To further improve the distinctiveness of captioning models, we propose a simple and effective training strategy that trains the model by comparing target i…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）；Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）
Large-scale vision-language pre-training has achieved promising results on downstream tasks. Existing methods highly rely on the assumption that the image-text pairs crawled from the Internet are in perfect one-to-one correspondence. However, in real scenarios, this assumption can be difficult to hold: the text description, obtained by crawling the affiliated metadata of the image, often suffers from the semantic mismatch and the mutual compatibility. To address these issues, we introduce PyramidCLIP, which constructs an input pyramid with different semantic levels for each modality, and align…
FineCLIPER: Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERs
School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；The University of Hong Kong（香港大学）；Nanyang Technological University（南洋理工大学）；School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；Unmanned System Research Institute, Northwestern Polytechnical University（西北工业大学无人系统研究所）
Detecting AI-Generated Images via Distributional Deviations from Real Images
Henan Key Laboratory of Big Data Analysis and Processing, School of Computer and Information Engineering, Henan University（河南大数据分析与处理重点实验室，计算机与信息工程学院，河南大学）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Peng Cheng Laboratory（鹏城实验室）；University of Chinese Academy of Sciences（中国科学院大学）
Although significant progress has been made in few-shot learning, most of existing few-shot image classification methods require supervised pre-training on a large amount of samples of base classes, which limits their generalization ability in real world application. Recently, large-scale Vision-Language Pre-trained models (VLPs) have been gaining increasing attention in few-shot learning because they can provide a new paradigm for transferable visual representation learning with easily available text on the Web. However, the VLPs may neglect detailed visual information that is difficult to de…
Evaluating and comparing text-to-image models is a challenging problem. Significant advances in the field have recently been made, piquing interest of various industrial sectors. As a consequence, a gold standard in the field should cover a variety of tasks and application contexts. In this paper a novel evaluation approach is experimented, on the basis of: (i) a curated data set, made by high-quality royalty-free image-text pairs, divided into ten categories; (ii) a quantitative metric, the CLIP-score, (iii) a human evaluation task to distinguish, for a given text, the real and the generated …
Will It Zero-Shot?: Predicting Zero-Shot Classification Performance For Arbitrary Queries
Computer Science George Washington University Washington, DC, USA（计算机科学 华盛顿大学 华盛顿特区, 美国）；Computer Science Saint Louis University Saint Louis, USA（计算机科学 圣路易斯大学 圣路易斯, 美国）
RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognition
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；MThreads, Inc.（MThreads公司）；Nanyang Technological University（南洋理工大学）
Micro-gesture Online Recognition using Learnable Query Points
School of Computer Science and Information Engineering（计算机与信息工程学院）；School of Artificial Intelligence（人工智能学院）；Hefei University of Technology (HFUT)（合肥工业大学）；Key Laboratory of Knowledge Engineering with Big Data (HFUT), Ministry of Education（大数据知识工程教育部重点实验室）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Zhejiang University（浙江大学）；Anhui Zhonghuitong Technology Co., Ltd.（安徽中汇通科技有限公司）
DeeCLIP: A Robust and Generalizable Transformer-Based Framework for Detecting AI-Generated Images
Laboratory of IEMN, Univ. Polytechnique Hauts-de-France（里莫恩实验室，法国高等技术大学）；Khalifa University（卡利法大学）；Sorbonne Center for Artificial Intelligence, Sorbonne University（索邦人工智能中心，索邦大学）
Multi-proposal Collaboration and Multi-task Training for Weakly-supervised Video Moment Retrieval
College of Computer Science and Electronic Engineering（计算机科学与电子工程学院）；Faculty of Electronic Engineering and Computer Science（电子工程与计算机科学系）；Mathematical and Data Science Center（数学与数据科学中心）；Graduate School of Informatics（信息科学研究生院）
Cross-modality interaction is a critical component in Text-Video Retrieval (TVR), yet there has been little examination of how different influencing factors for computing interaction affect performance. This paper first studies the interaction paradigm in depth, where we find that its computation can be split into two terms, the interaction contents at different granularity and the matching function to distinguish pairs with the same semantics. We also observe that the single-vector representation and implicit intensive function substantially hinder the optimization. Based on these findings, w…
