Precise video retrieval requires multi-modal correlations to handle unseen vocabulary and scenes, becoming more complex for lengthy videos where models must perform effectively without prior training on a specific dataset. We introduce a unified framework that combines a visual matching stream and an aural matching stream with a unique subtitles-based video segmentation approach. Additionally, the aural stream includes a complementary audio-based two-stage retrieval mechanism that enhances performance on long-duration videos. Considering the complex nature of retrieval from lengthy videos and its corresponding evaluation, we introduce a new retrieval evaluation method specifically designed for long-video retrieval to support further research. We conducted experiments on the YouCook2 benchmark, showing promising retrieval performance.
ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge
College of Computer Science, Sichuan University, Chengdu, China（四川大学计算机学院）；National Key Laboratory of Fundamental Algorithms and Models for Engineering Simulation, Sichuan University, China（四川省工程仿真基础算法与模型国家重点实验室）
V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrieval
Tsinghua University（清华大学）；University of Central Florida（中央佛罗里达大学）；Fudan University（复旦大学）；The Australian National University（澳大利亚国立大学）；The University of Hong Kong（香港大学）；Pengcheng Laboratory（鹏城实验室）；Central South University（中南大学）
Hashing has been widely applied to multimodal retrieval on large-scale multimedia data due to its efficiency in computation and storage. In this article, we propose a novel deep semantic multimodal hashing network (DSMHN) for scalable image-text and video-text retrieval. The proposed deep hashing framework leverages 2-D convolutional neural networks (CNN) as the backbone network to capture the spatial information for image-text retrieval, while the 3-D CNN as the backbone network to capture the spatial and temporal information for video-text retrieval. In the DSMHN, two sets of modality-specif…
AdaptiveEmbed: Sample-Adaptive Multi-Vector Representation for Multimodal Retrieval
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
Eurecat, Centre Tecnològic de Catalunya（加泰罗尼亚技术中心）；Computer Vision Center, Universitat Autònoma de Barcelona（巴塞罗那自治大学计算机视觉中心）
Self-Supervised learning from multimodal image and text data allows deep neural networks to learn powerful features with no need of human annotated data. Web and Social Media platforms provide a virtually unlimited amount of this multimodal data. In this work we propose to exploit this free available data to learn a multimodal image and text embedding, aiming to leverage the semantic knowledge learnt in the text domain and transfer it to a visual model for semantic image retrieval. We demonstrate that the proposed pipeline can learn from images with associated textwithout supervision and analy…
MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents
INSAIT；Lanzhou University（兰州大学）；King Abdullah University of Science and Technology（国王阿卜杜勒-阿齐兹科学与技术大学）；Heriot-Watt University（赫瑞-沃德大学）；Monash University（莫纳什大学）；University College Dublin（都柏林大学学院）
Brain-Inspired Multimodal Spiking Neural Network for Image-Text Retrieval
Hubei Key Laboratory of Transportation Internet of Things, Wuhan University of Technology（武汉理工大学交通物联网湖北省重点实验室）；State Key Laboratory for Multimedia Information Processing, Peking University（北京大学多媒体信息处理全国重点实验室）
Reasoning Guided Embeddings: Leveraging MLLM Reasoning for Improved Multimodal Retrieval
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；Sensetime Research（商汤科技研究院）；Beijing Institute of Technology（北京理工大学）；Shanghai AI Lab（上海人工智能实验室）
Cross-modal image-recipe retrieval has gained significant attention in recent years. Most work focuses on improving cross-modal embeddings using unimodal encoders, that allow for efficient retrieval in large-scale databases, leaving aside cross-attention between modalities which is more computationally expensive. We propose a new retrieval framework, T-Food (Transformer Decoders with MultiModal Regularization for Cross-Modal Food Retrieval) that exploits the interaction between modalities in a novel regularization scheme, while using only unimodal encoders at test time for efficient retrieval.…
In this work we present a new State-of-The-Art on the text-to-video retrieval task on MSR-VTT, LSMDC, MSVD, YouCook2 and TGIF obtained by a single model. Three different data sources are combined: weakly-supervised videos, crowd-labeled text-image pairs and text-video pairs. A careful analysis of available pre-trained networks helps to choose the best prior-knowledge ones. We introduce three-stage training procedure that provides high transfer knowledge efficiency and allows to use noisy datasets during training without prior knowledge degradation. Additionally, double positional encoding is u…
Lomonosov Moscow State University（莫斯科国立大学）；Huawei Moscow Research Center（华为莫斯科研究中心）
We present a new state-of-the-art on the text to video retrieval task on MSRVTT and LSMDC benchmarks where our model outperforms all previous solutions by a large margin. Moreover, state-of-the-art results are achieved with a single model on two datasets without finetuning. This multidomain generalisation is achieved by a proper combination of different video caption datasets. We show that training on different datasets can improve test results of each other. Additionally we check intersection between many popular datasets and found that MSRVTT has a significant overlap between the test and th…
In tissue characterization and cancer diagnostics, multimodal imaging has emerged as a powerful technique. Thanks to computational advances, large datasets can be exploited to discover patterns in pathologies and improve diagnosis. However, this requires efficient and scalable image retrieval methods. Cross-modality image retrieval is particularly challenging, since images of similar (or even the same) content captured by different modalities might share few common structures. We propose a new application-independent content-based image retrieval (CBIR) system for reverse (sub-)image search ac…
This paper addresses the problem of media retrieval using a multimodal query (a query which combines visual input with additional semantic information in natural language feedback). We propose a SynthTriplet GAN framework which resolves this task by expanding the multimodal query with a synthetically generated image that captures semantic information from both image and text input. We introduce a novel triplet mining method that uses a synthetic image as an anchor to directly optimize for embedding distances of generated and target images. We demonstrate that apart from the added value of retr…
University of Tübingen（图宾根大学）；MPI for Informatics（马克斯·普朗克信息学研究所）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）
Existing works in image retrieval often consider retrieving images with one or two query inputs, which do not generalize to multiple queries. In this work, we investigate a more challenging scenario for composing multiple multimodal queries in image retrieval. Given an arbitrary number of query images and (or) texts, our goal is to retrieve target images containing the semantic concepts specified in multiple multimodal queries. To learn an informative embedding that can flexibly encode the semantics of various queries, we propose a novel multimodal probabilistic composer (MPC). Specifically, w…
BRIDGE: Multimodal-to-Text Retrieval via Reinforcement-Learned Query Alignment
High institute for computer & information systems（高等计算机与信息系统学院）；Chungbuk National University（忠北大学）；Assiut University（艾斯尤特大学）；University of Innsbruck（因斯布鲁克大学）
We propose a novel, efficient, modular and scalable framework for content based visual media retrieval systems by leveraging the power of Deep Learning which is flexible to work both for images and videos conjointly and we also introduce an efficient comparison and filtering metric for retrieval. We put forward our findings from critical performance tests comparing our method to the predominant conventional approach to demonstrate the feasibility and efficiency of the proposed solution with best practices, possible improvements that may further augment the ability of retrieval architectures.
Most existing image retrieval systems use text queries as a way for the user to express what they are looking for. However, fine-grained image retrieval often requires the ability to also express where in the image the content they are looking for is. The text modality can only cumbersomely express such localization preferences, whereas pointing is a more natural fit. In this paper, we propose an image retrieval setup with a new form of multimodal queries, where the user simultaneously uses both spoken natural language (the what) and mouse traces over an empty canvas (the where) to express the…
We introduce CommerceMM - a multimodal model capable of providing a diverse and granular understanding of commerce topics associated to the given piece of content (image, text, image+text), and having the capability to generalize to a wide range of tasks, including Multimodal Categorization, Image-Text Retrieval, Query-to-Product Retrieval, Image-to-Product Retrieval, etc. We follow the pre-training + fine-tuning training regime and present 5 effective pre-training tasks on image-text pairs. To embrace more common and diverse commerce data with text-to-multimodal, image-to-multimodal, and mult…
National School of Engineers (ENIS)（国立工程师学院（ENIS））
In this paper, we present an overview of a multimodal system to indexing and searching video sequence by the content that has been developed within the REGIMVid project. A large part of our system has been developed as part of TRECVideo evaluation. The MAVSIR platform provides High-level feature extraction from audio-visual content and concept/event-based video retrieval. We illustrate the architecture of the system as well as provide an overview of the descriptors supported to date. Then we demonstrate the usefulness of the toolbox in the context of feature extraction, concepts/events learnin…
RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understanding
University of Copenhagen（哥本哈根大学）；ETH Zürich（苏黎世联邦理工学院）；University of Amsterdam（阿姆斯特丹大学）；University of Cambridge（剑桥大学）；Massachusetts Institute of Technology（麻省理工学院）；Singapore University of Technology and Design（新加坡科技设计大学）；Singapore Management University（新加坡管理大学）
Beyond Classification: Dynamic Adapter Routing for Continual Multimodal Retrieval
NASK National Research Institute（NASK国家研究院）；IDEAS Research Institute（IDEAS研究所）；Warsaw University of Technology（华沙技术大学）；Universitat Autonoma de Barcelona（巴塞罗那自治大学）
SnapBench: Benchmarking Snap-and-Ask Multimodal Retrieval for Mobile Interactions
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Tencent Yuanbao（腾讯元宝）；Tsinghua University（清华大学）；ARC Lab, Tencent（腾讯ARC实验室）；The University of Hong Kong（香港大学）
Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS）；Alibaba Cloud Computing（阿里巴巴云计算）
Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrieval
Shanghai Jiao Tong University（上海交通大学）；Institute of Digital Twin, Eastern Institute of Technology（数字孪生研究院，东部技术研究所）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative（宁波空间智能与数字衍生关键实验室）
