Person search by natural language aims at retrieving a specific person in a large-scale image pool that matches the given textual descriptions. While most of the current methods treat the task as a holistic visual and textual feature matching one, we approach it from an attribute-aligning perspective that allows grounding specific attribute phrases to the corresponding visual regions. We achieve success as well as the performance boosting by a robust feature learning that the referred identity can be accurately bundled by multiple attribute visual cues. To be concrete, our Visual-Textual Attribute Alignment model (dubbed as ViTAA) learns to disentangle the feature space of a person into subspaces corresponding to attributes using a light auxiliary attribute segmentation computing branch. It then aligns these visual features with the textual attributes parsed from the sentences by using a novel contrastive learning loss. Upon that, we validate our ViTAA framework through extensive experiments on tasks of person search by natural language and by attribute-phrase queries, on which our system achieves state-of-the-art performances. Code will be publicly available upon publication.
Searching persons in large-scale image databases with the query of natural language description is a more practical important applications in video surveillance. Intuitively, for person search, the core issue should be visual-textual association, which is still an extremely challenging task, due to the contradiction between the high abstraction of textual description and the intuitive expression of visual images. However, for this task, while positive image-text pairs are always well provided, most existing methods doesn't tackle this problem effectively by mining more reasonable negative pair…
Prompt-driven Transferable Adversarial Attack on Person Re-Identification with Attribute-aware Textual Inversion
School of Artificial Intelligence and Robotics, Hunan University（湖南大学人工智能与机器人学院）；National Engineering Research Center of Robot Visual Perception and Control Technology（机器人视觉感知与控制技术国家工程研究中心）；College of Information Science and Engineering, Hunan Normal University（湖南师范大学信息科学与工程学院）
Tencent YouTu Lab（腾讯优图实验室）；AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院）
Text-based Person Search (TPS), is targeted on retrieving pedestrians to match text descriptions instead of query images. Recent Vision-Language Pre-training (VLP) models can bring transferable knowledge to downstream TPS tasks, resulting in more efficient performance gains. However, existing TPS methods improved by VLP only utilize pre-trained visual encoders, neglecting the corresponding textual representation and breaking the significant modality alignment learned from large-scale pre-training. In this paper, we explore the full utilization of textual potential from VLP in TPS tasks. We bui…
Text-based person retrieval aims to find the query person based on a textual description. The key is to learn a common latent space mapping between visual-textual modalities. To achieve this goal, existing works employ segmentation to obtain explicitly cross-modal alignments or utilize attention to explore salient alignments. These methods have two shortcomings: 1) Labeling cross-modal alignments are time-consuming. 2) Attention methods can explore salient cross-modal alignments but may ignore some subtle and valuable pairs. To relieve these issues, we introduce an Implicit Visual-Textual (IVT…
Universidade Federal de Minas Gerais (UFMG)（米纳斯吉拉斯联邦大学）
The growth of Social Networks has fueled the habit of people logging their day-to-day activities, and long First-Person Videos (FPVs) are one of the main tools in this new habit. Semantic-aware fast-forward methods are able to decrease the watch time and select meaningful moments, which is key to increase the chances of these videos being watched. However, these methods can not handle semantics in terms of personalization. In this work, we present a new approach to automatically creating personalized fast-forward videos for FPVs. Our approach explores the availability of text-centric data from…
Text-to-image person retrieval aims to identify the target person based on a given textual description query. The primary challenge is to learn the mapping of visual and textual modalities into a common latent space. Prior works have attempted to address this challenge by leveraging separately pre-trained unimodal models to extract visual and textual features. However, these approaches lack the necessary underlying alignment capabilities required to match multimodal data effectively. Besides, these works use prior information to explore explicit part alignments, which may lead to the distortio…
ShapeSpeak: Body Shape-Aware Textual Alignment for Visible-Infrared Person Re-Identification
Nanjing University of Science and Technology（南京理工大学）；Chongqing University of Posts and Telecommunications（重庆邮电大学）；The Hong Kong Polytechnic University（香港理工大学）
Dynamic Textual Prompt For Rehearsal-free Lifelong Person Re-identification
National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology（空天地海一体化大数据应用技术国家工程实验室）；School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）
Contextualized Visual Personalization in Vision-Language Models
Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea（电气电子工程系，首尔国立大学，首尔，韩国）；Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, Korea（人工智能交叉学科项目，首尔国立大学，首尔，韩国）
South China University of Technology（华南理工大学）；Pazhou Lab（琶洲实验室）；JD Explore Academy at JD.com（京东集团京东探索研究院）
Text-to-image person re-identification (ReID) aims to search for images containing a person of interest using textual descriptions. However, due to the significant modality gap and the large intra-class variance in textual descriptions, text-to-image ReID remains a challenging problem. Accordingly, in this paper, we propose a Semantically Self-Aligned Network (SSAN) to handle the above problems. First, we propose a novel method that automatically extracts semantically aligned part-level features from the two modalities. Second, we design a multi-view non-local network that captures the relatio…
Recent advancement of research in biometrics, computer vision, and natural language processing has discovered opportunities for person retrieval from surveillance videos using textual query. The prime objective of a surveillance system is to locate a person using a description, e.g., a short woman with a pink t-shirt and white skirt carrying a black purse. She has brown hair. Such a description contains attributes like gender, height, type of clothing, colour of clothing, hair colour, and accessories. Such attributes are formally known as soft biometrics. They help bridge the semantic gap betw…
Nanjing University of information science and Technology（南京信息工程大学）；Xinjiang University（新疆大学）；Nanjing Institute of Technology（南京工程学院）；KingSaud University（沙特国王大学）
Text-based person re-identification(Re-id) is an important task in video surveillance, which consists of retrieving the corresponding person's image given a textual description from a large gallery of images. It is difficult to directly match visual contents with the textual descriptions due to the modality heterogeneity. On the one hand, the textual embeddings are not discriminative enough, which originates from the high abstraction of the textual descriptions. One the other hand,Global average pooling (GAP) is commonly utilized to extract more general or smoothed features implicitly but igno…
The Chinese University of Hong Kong（香港中文大学）；Massachuate Institute of Technology（麻省理工学院）；SenseTime Group Limited（商汤集团有限公司）
Searching persons in large-scale image databases with the query of natural language description has important applications in video surveillance. Existing methods mainly focused on searching persons with image-based or attribute-based queries, which have major limitations for a practical usage. In this paper, we study the problem of person search with natural language description. Given the textual description of a person, the algorithm of the person search is required to rank all the samples in the person database then retrieve the most relevant sample corresponding to the queried description…
Harnessing the Power of MLLMs for Transferable Text-to-Image Person ReID
South China University of Technology（华南理工大学）；Pazhou Lab（琶洲实验室）；JD Explore Academy（京东探索研究院）；Yunnan University（云南大学）；Yunnan United Vision Technology Co., Ltd.（云南联慧视觉科技有限公司）
South China University of Technology（华南理工大学）；Baidu VIS（百度视觉技术部）；University of Liverpool（利物浦大学）
Text-to-image person re-identification (ReID) aims to search for pedestrian images of an interested identity via textual descriptions. It is challenging due to both rich intra-modal variations and significant inter-modal gaps. Existing works usually ignore the difference in feature granularity between the two modalities, i.e., the visual features are usually fine-grained while textual features are coarse, which is mainly responsible for the large inter-modal gaps. In this paper, we propose an end-to-end framework based on transformers to learn granularity-unified representations for both modal…
Southern University of Science and Technology（南方科技大学）；Tencent Youtu Lab（腾讯优图实验室）
Text-based person search aims at retrieving target person in an image gallery using a descriptive sentence of that person. It is very challenging since modal gap makes effectively extracting discriminative features more difficult. Moreover, the inter-class variance of both pedestrian images and descriptions is small. So comprehensive information is needed to align visual and textual clues across all scales. Most existing methods merely consider the local alignment between images and texts within a single scale (e.g. only global scale or only partial scale) then simply construct alignment at ea…
Text-to-image models offer unprecedented freedom to guide creation through natural language. Yet, it is unclear how such freedom can be exercised to generate images of specific unique concepts, modify their appearance, or compose them in new roles and novel scenes. In other words, we ask: how can we use language-guided models to turn our cat into a painting, or imagine a new product based on our favorite toy? Here we present a simple approach that allows such creative freedom. Using only 3-5 images of a user-provided concept, like an object or a style, we learn to represent it through new "wor…
Nanjing University of Information Science and Technology（南京信息工程大学）；Massey University（梅西大学）
Text-based person search is a sub-task in the field of image retrieval, which aims to retrieve target person images according to a given textual description. The significant feature gap between two modalities makes this task very challenging. Many existing methods attempt to utilize local alignment to address this problem in the fine-grained level. However, most relevant methods introduce additional models or complicated training and evaluation strategies, which are hard to use in realistic scenarios. In order to facilitate the practical application, we propose a simple but effective end-to-en…
DiCo: Disentangled Concept Representation for Text-to-image Person Re-identification
organization= Department of Imaging Science, Graduate School of Advanced Imaging Science, Multimedia \& Film, Chung-Ang University , addressline= , city= Seoul , postcode= 06974 , state= , country= South Korea；organization= Department of Metaverse Convergence, Graduate School of Advanced Imaging Science, Multimedia \& Film, Chung-Ang University , addressline= , city= Seoul , postcode= 06974 , state= , country= South Korea
Text-guided Image Restoration and Semantic Enhancement for Text-to-Image Person Retrieval
School of Artificial Intelligence（人工智能学院）；Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing Key Laboratory of Network System and Network Culture（网络系统与网络文化北京市重点实验室）
SCMM: Calibrating Cross-modal Representations for Text-Based Person Search
College of Future Information Technology, Fudan University（复旦大学未来信息技术学院）；Department of Electrical and Computer Engineering, The University of British Columbia（不列颠哥伦比亚大学电气与计算机工程系）；College of Electronic and Information Engineering, Tongji University（同济大学电子与信息工程学院）；MEGVII Technology（MEGVII技术）；School of Computer Science and Informatics, Cardiff University（卡迪夫大学计算机科学与信息学院）；School of AI and CS, Nantong University（南通大学人工智能与计算机科学学院）；Academy of Artificial Intelligence, SMBU（SMBU人工智能学院）；College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）
Text-based person search aims to retrieve images of a certain pedestrian by a textual description. The key challenge of this task is to eliminate the inter-modality gap and achieve the feature alignment across modalities. In this paper, we propose a semantic-aligned embedding method for text-based person search, in which the feature alignment across modalities is achieved by automatically learning the semantic-aligned visual features and textual features. First, we introduce two Transformer-based backbones to encode robust feature representations of the images and texts. Second, we design a se…
CILP-FGDI: Exploiting Vision-Language Model for Generalizable Person Re-Identification
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education（教育部新一代人工智能技术及其交叉应用重点实验室（东南大学））
Cross-modal Fuzzy Alignment Network for Text-Aerial Person Retrieval and A Large-scale Benchmark
State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology（光电信息采集与防护技术国家重点实验室）；School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；The University of Hong Kong（香港大学）
ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models
Institute of Automation, CAS（中国科学院自动化研究所）；School of Artificial Intelligence, UCAS（中国科学院大学人工智能学院）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）；Kuaishou Technology（快手科技）；National Cheng-Kung University（国立成功大学）；University of Konstanz（康斯坦茨大学）
Automatic forensic image analysis assists criminal investigation experts in the search for suspicious persons, abnormal behaviors detection and identity matching in images. In this paper we propose a person retrieval system that uses textual queries (e.g., "black trousers and green shirt") as descriptions and a one-class generative color model with outlier filtering to represent the images both to train the models and to perform the search. The method is evaluated in terms of its efficiency in fulfilling the needs of a forensic retrieval system: limited annotation, robustness, extensibility, a…
Generative Networks have proved to be extremely effective in image restoration and reconstruction in the past few years. Generating faces from textual descriptions is one such application where the power of generative algorithms can be used. The task of generating faces can be useful for a number of applications such as finding missing persons, identifying criminals, etc. This paper discusses a novel approach to generating human faces given a textual description regarding the facial features. We use the power of state of the art natural language processing models to convert face descriptions i…
HopaDIFF: Holistic-Partial Aware Fourier Conditioned Diffusion for Referring Human Action Segmentation in Multi-Person Scenarios
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Beijing Institute of Technology（北京理工大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Hunan University（湖南大学）；Shanghai AI Lab（上海人工智能实验室）；HEBUST
University of Surrey（萨里大学）；Surrey Institute for People-centred AI（萨里以人为本人工智能研究所）；Sensus Futuris Ltd.（Sensus Futuris有限公司）
Cross-modal person re-identification (Re-ID) is critical for modern video surveillance systems. The key challenge is to align cross-modality representations induced by the semantic information present for a person and ignore background information. This work presents a novel convolutional neural network (CNN) based architecture designed to learn semantically aligned cross-modal visual and textual representations. The underlying building block, named AXM-Block, is a unified multi-layer network that dynamically exploits the multi-scale knowledge from both modalities and re-calibrates each modali…
Sentiment analysis is attracting more and more attentions and has become a very hot research topic due to its potential applications in personalized recommendation, opinion mining, etc. Most of the existing methods are based on either textual or visual data and can not achieve satisfactory results, as it is very hard to extract sufficient information from only one single modality data. Inspired by the observation that there exists strong semantic correlation between visual and textual data in social medias, we propose an end-to-end deep fusion convolutional neural network to jointly learn text…
Generating coherent and useful image/video scenes from a free-form textual description is technically a very difficult problem to handle. Textual description of the same scene can vary greatly from person to person, or sometimes even for the same person from time to time. As the choice of words and syntax vary while preparing a textual description, it is challenging for the system to reliably produce a consistently desirable output from different forms of language input. The prior works of scene generation have been mostly confined to rigorous sentence structures of text input which restrict t…
Automatic Synthetic Data and Fine-grained Adaptive Feature Alignment for Composed Person Retrieval
Beijing University of Posts and Telecommunications（北京邮电大学）；SenseTime（商汤科技）；Beijing Key Laboratory of Network System and Network Culture（北京网络系统与网络文化重点实验室）；Key Laboratory of Intereactive Technology and Experience System, Ministry of Culture and Tourism（文化和旅游部交互技术与体验系统重点实验室）
The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）
Referring expression grounding aims at locating certain objects or persons in an image with a referring expression, where the key challenge is to comprehend and align various types of information from visual and textual domain, such as visual attributes, location and interactions with surrounding regions. Although the attention mechanism has been successfully applied for cross-modal alignments, previous attention models focus on only the most dominant features of both modalities, and neglect the fact that there could be multiple comprehensive textual-visual correspondences between images and r…
