This paper addresses the problem of media retrieval using a multimodal query (a query which combines visual input with additional semantic information in natural language feedback). We propose a SynthTriplet GAN framework which resolves this task by expanding the multimodal query with a synthetically generated image that captures semantic information from both image and text input. We introduce a novel triplet mining method that uses a synthetic image as an anchor to directly optimize for embedding distances of generated and target images. We demonstrate that apart from the added value of retrieval illustration with synthetic image with the focus on customization and user feedback, the proposed method greatly surpasses other multimodal generation methods and achieves state of the art results in the multimodal retrieval task. We also show that in contrast to other retrieval methods, our method provides explainable embeddings.
Beijing Institute of Technology（北京理工大学）；JD AI Research（京东人工智能研究院）；Australian National University（澳大利亚国立大学）
Multimodal named entity recognition (MNER) is a critical step in information extraction, which aims to detect entity spans and classify them to corresponding entity types given a sentence-image pair. Existing methods either (1) obtain named entities with coarse-grained visual clues from attention mechanisms, or (2) first detect fine-grained visual regions with toolkits and then recognize named entities. However, they suffer from improper alignment between entity types and visual regions or error propagation in the two-stage manner, which finally imports irrelevant visual information into texts…
BAS-OPD: Budget-Aware Selective On-Policy Self-Distillation for Fine-Grained Multimodal Perception
HFIPS, Chinese Academy of Sciences（中国科学院合肥物质科学研究院）；University of Science and Technology of China（中国科学技术大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
BRIDGE: Multimodal-to-Text Retrieval via Reinforcement-Learned Query Alignment
High institute for computer & information systems（高等计算机与信息系统学院）；Chungbuk National University（忠北大学）；Assiut University（艾斯尤特大学）；University of Innsbruck（因斯布鲁克大学）
We propose a new spatial memory module and a spatial reasoner for the Visual Grounding (VG) task. The goal of this task is to find a certain object in an image based on a given textual query. Our work focuses on integrating the regions of a Region Proposal Network (RPN) into a new multi-step reasoning model which we have named a Multimodal Spatial Region Reasoner (MSRR). The introduced model uses the object regions from an RPN as initialization of a 2D spatial memory and then implements a multi-step reasoning process scoring each region according to the query, hence why we call it a multimodal…
Autoregressive Retriever: Improving Query Understanding from Item Feedback for Universal Multimodal Retrieval
School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；Zhongguancun Academy（中关村实验室）；Xiaohongshu（小红书）；Southeast Academy of Information Technology, Beijing Institute of Technology（北京理工大学东南信息科学研究院）；Zhongguancun Laboratory（中关村实验室）
Bridging Pixels and Words: Mask-Aware Local Semantic Fusion for Multimodal Media Verification
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所人机混合增强智能全国重点实验室）
UniRef-UAV: A Multimodal Benchmark for Universal Referring in UAV Imagery
School of Automation, Northwestern Polytechnical University（自动化学院，西北工业大学）；College of Missile Engineering, Rocket Force University of Engineering（导弹工程学院，火箭兵工程大学）；School of Artificial Intelligence, Chongqing University of Posts and Telecommunications（人工智能学院，重庆邮电大学）
Multi-Grained Query-Guided Set Prediction Network for Grounded Multimodal Named Entity Recognition
School of Artificial Intelligence, Sun Yat-sen University（中山大学人工智能学院）；School of Atmospheric Sciences, Sun Yat-sen University（中山大学大气科学学院）；School of Electronics and Communication Engineering, Sun Yat-sen University（中山大学电子与通信工程学院）；State Key Laboratory of Intelligent Game, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所智能游戏国家重点实验室）；Pazhou Lab（琶洲实验室）
University of Tübingen（图宾根大学）；MPI for Informatics（马克斯·普朗克信息学研究所）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）
Existing works in image retrieval often consider retrieving images with one or two query inputs, which do not generalize to multiple queries. In this work, we investigate a more challenging scenario for composing multiple multimodal queries in image retrieval. Given an arbitrary number of query images and (or) texts, our goal is to retrieve target images containing the semantic concepts specified in multiple multimodal queries. To learn an informative embedding that can flexibly encode the semantics of various queries, we propose a novel multimodal probabilistic composer (MPC). Specifically, w…
Vector Institute for AI（向量人工智能研究所）；Meta AI；University of British Columbia（不列颠哥伦比亚大学）
Video understanding tasks take many forms, from action detection to visual query localization and spatio-temporal grounding of sentences. These tasks differ in the type of inputs (only video, or video-query pair where query is an image region or sentence) and outputs (temporal segments or spatio-temporal tubes). However, at their core they require the same fundamental understanding of the video, i.e., the actors and objects in it, their actions and interactions. So far these tasks have been tackled in isolation with individual, highly specialized architectures, which do not exploit the interpl…
Seeing the Forest and the Trees: Query-Aware Tokenizer for Long-Video Multimodal Language Models
Queen Mary University of London（伦敦女王学院）；University of Sheffield（谢菲尔德大学）；Imperial College London（伦敦帝国学院）；Pengcheng Laboratory（鹏城实验室）；Meta Inc（Meta公司）；Meituan Inc（美团公司）；Nanjing University of Science（南京理工大学）；University of Birmingham（伯明翰大学）；Utrecht University（乌得勒支大学）
TAP-RAG: Task-Aware Policy Control for Long-Document Multimodal Question Answering
School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；The International Joint Institute of Tianjin University（天津大学国际联合学院）；Department of Automation, Tsinghua University（清华大学自动化系）
To Be Multimodal or Not to Be: Query-Adaptive Audio-Visual Person Retrieval via Active Modality Detection
University of Cambridge（剑桥大学）；Queen's University Belfast（贝尔法斯特女王大学）；University of Surrey（萨里大学）；Cisco（思科）；Southwest Jiaotong University（西南交通大学）；Teesside University（泰赛德大学）
Task-Oriented Visual Feature Compression via Residual Vector Quantization for Device-Edge Multimodal Inference
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；China Telecom（中国电信）；Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））
We propose a new zero-shot Event Detection method by Multi-modal Distributional Semantic embedding of videos. Our model embeds object and action concepts as well as other available modalities from videos into a distributional semantic space. To our knowledge, this is the first Zero-Shot event detection model that is built on top of distributional semantics and extends it in the following directions: (a) semantic embedding of multimodal information in videos (with focus on the visual modalities), (b) automatically determining relevance of concepts/attributes to a free text query, which could be…
In this paper, we address the problem of spatio-temporal person retrieval from multiple videos using a natural language query, in which we output a tube (i.e., a sequence of bounding boxes) which encloses the person described by the query. For this problem, we introduce a novel dataset consisting of videos containing people annotated with bounding boxes for each second and with five natural language descriptions. To retrieve the tube of the person described by a given natural language query, we design a model that combines methods for spatio-temporal human detection and multimodal retrieval. W…
Semantic-Spatial Agreement Verification for Mitigating Object Hallucination in Multimodal Large Language Models
Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））；Shandong Computer Science Center (National Supercomputer Center in Jinan)（山东省计算中心（国家超级计算济南中心））；China Telecom Digital Intelligence Technology Co., Ltd.（中国电信数字智能科技有限公司）；Shenyang Aerospace University（沈阳航空航天大学）
East China Normal University（华东师范大学）；Alibaba Group（阿里巴巴集团）；NLPR, CASIA（中国科学院自动化研究所模式识别国家重点实验室）；Fudan University（复旦大学）
Visual grounding focuses on establishing fine-grained alignment between vision and natural language, which has essential applications in multimodal reasoning systems. Existing methods use pre-trained query-agnostic visual backbones to extract visual feature maps independently without considering the query information. We argue that the visual features extracted from the visual backbones and the features really needed for multimodal reasoning are inconsistent. One reason is that there are differences between pre-training tasks and visual grounding. Moreover, since the backbones are query-agnost…
How would you search for a unique, fashionable shoe that a friend wore and you want to buy, but you didn't take a picture? Existing approaches propose interactive image search as a promising venue. However, they either entrust the user with taking the initiative to provide informative feedback, or give all control to the system which determines informative questions to ask. Instead, we propose a mixed-initiative framework where both the user and system can be active participants, depending on whose initiative will be more beneficial for obtaining high-quality search results. We develop a reinf…
Multimodal Reasoning Agent for Zero-Shot Composed Image Retrieval
College of Computing and Data Science, Nanyang Technological University, Singapore（南洋理工大学计算机与数据科学学院）；School of Information Science and Technology, University of Science and Technology of China, Hefei, China（中国科学技术大学信息科学与技术学院）；Sun Yat-sen University Shenzhen Campus, School of Cyber Science and Technology, Shenzhen, China（中山大学深圳校区计算机科学与技术学院）
