We address the challenges inherent in sketch-based image retrieval (SBIR) across various settings, including zero-shot SBIR, generalized zero-shot SBIR, and fine-grained zero-shot SBIR, by leveraging the vision-language foundation model CLIP. While recent endeavors have employed CLIP to enhance SBIR, these approaches predominantly follow uni-modal prompt processing and overlook to exploit CLIP's integrated visual and textual capabilities fully. To bridge this gap, we introduce SpLIP, a novel multi-modal prompt learning scheme designed to operate effectively with frozen CLIP backbones. We diverge from existing multi-modal prompting methods that treat visual and textual prompts independently or integrate them in a limited fashion, leading to suboptimal generalization. SpLIP implements a bi-directional prompt-sharing strategy that enables mutual knowledge exchange between CLIP's visual and textual encoders, fostering a more cohesive and synergistic prompt processing mechanism that significantly reduces the semantic gap between the sketch and photo embeddings. In addition to pioneering multi-modal prompt learning, we propose two innovative strategies for further refining the embedding space. The first is an adaptive margin generation for the sketch-photo triplet loss, regulated by CLIP's class textual embeddings. The second introduces a novel task, termed conditional cross-modal jigsaw, aimed at enhancing fine-grained sketch-photo alignment by implicitly modeling sketches' viable patch arrangement using knowledge of unshuffled photos. Our comprehensive experimental evaluations across multiple benchmarks demonstrate the superior performance of SpLIP in all three SBIR scenarios. Project page: https://mainaksingha01.github.io/SpLIP/ .
How would you search for a unique, fashionable shoe that a friend wore and you want to buy, but you didn't take a picture? Existing approaches propose interactive image search as a promising venue. However, they either entrust the user with taking the initiative to provide informative feedback, or give all control to the system which determines informative questions to ask. Instead, we propose a mixed-initiative framework where both the user and system can be active participants, depending on whose initiative will be more beneficial for obtaining high-quality search results. We develop a reinf…
Bridging the Modality Gap in Forensic Image Retrieval
Advanced Technologies Application Center (CENATAV)（先进技术应用中心（CENATAV））；Centro de Sistemas Complejos, Facultad de Física, Universidad de La Habana（哈瓦那大学物理学院复杂系统中心）
All in One Framework for Multimodal Re-identification in the Wild
National Engineering Research Center for Multimedia Software（国家多媒体软件工程技术研究中心）；Institute of Artificial Intelligence（人工智能研究院）；School of Computer Science（计算机学院）；Hubei Luojia Laboratory（湖北珞珈实验室）；Wuhan University（武汉大学）；Guangzhou Urban Planning Design Survey Research Institute（广州市城市规划勘测设计研究院）
FlexiReID: Adaptive Mixture of Expert for Multi-Modal Person Re-Identification
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（多媒体可信感知与高效计算重点实验室，中国教育部，厦门大学）；National University of Singapore（新加坡国立大学）；Tencent YouTu Lab（腾讯优图实验室）；Institute of Artificial Intelligence,Xiamen University（人工智能研究院，厦门大学）
When compared to unimodal systems, multimodal biometric systems have several advantages, including lower error rate, higher accuracy, and larger population coverage. However, multimodal systems have an increased demand for integrity and privacy because they must store multiple biometric traits associated with each user. In this paper, we present a deep learning framework for feature-level fusion that generates a secure multimodal template from each user's face and iris biometrics. We integrate a deep hashing (binarization) technique into the fusion architecture to generate a robust binary mult…
Existing conditional image synthesis frameworks generate images based on user inputs in a single modality, such as text, segmentation, sketch, or style reference. They are often unable to leverage multimodal user inputs when available, which reduces their practicality. To address this limitation, we propose the Product-of-Experts Generative Adversarial Networks (PoE-GAN) framework, which can synthesize images conditioned on multiple input modalities or any subset of them, even the empty set. PoE-GAN consists of a product-of-experts generator and a multimodal multiscale projection discriminator…
Whiting School of Engineering, Johns Hopkins University（约翰斯·霍普金斯大学怀廷工程学院）
Synthesis of face images from visual attributes is an important problem in computer vision and biometrics due to its applications in law enforcement and entertainment. Recent advances in deep generative networks have made it possible to synthesize high-quality face images from visual attributes. However, existing methods are specifically designed for generating unimodal images (i.e visible faces) from attributes. In this paper, we propose a novel generative adversarial network that simultaneously synthesizes identity preserving multimodal face images (i.e. visible, sketch, thermal, etc.) from …
Vision Also You Need: Navigating Out-of-Distribution Detection with Multimodal Large Language Model
Zhejiang University（浙江大学）；Tsinghua University（清华大学）；University of Chinese Academy of Sciences（中国科学院大学）；Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences)（教育部计算电力网络与信息安全重点实验室，山东计算机科学中心（国家超算中心济南中心），齐鲁工业大学（山东科学院））；Shandong Provincial Key Laboratory of Computing Power Internet and Service Computing, Shandong Fundamental Research Center for Computer Science（山东省计算电力互联网与服务计算重点实验室，山东省计算机科学基础研究中心）；University of Electronic Science and Technology of China（电子科技大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；RWTH Aachen University（亚琛工业大学）
AnyUser: Translating Sketched User Intent into Domestic Robots
College of Computer Science and Technology, National University of Defense Technology（国防科技大学计算机科学与技术学院）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（湖南大学国家机器人视觉感知与控制技术工程研究中心）
Multimodal-Conditioned Latent Diffusion Models for Fashion Image Editing
Media Integration and Communication Center (MICC), University of Florence（佛罗伦萨大学媒体集成与传播中心（MICC））；University of Florence（佛罗伦萨大学）；University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）；University of Pisa（比萨大学）
The key challenge in cross-modal retrieval is to find similarities between objects represented with different modalities, such as image and text. However, each modality embeddings stem from non-related feature spaces, which causes the notorious 'heterogeneity gap'. Currently, many cross-modal systems try to bridge the gap with self-attention. However, self-attention has been widely criticized for its quadratic complexity, which prevents many real-life applications. In response to this, we propose T-EMDE - a neural density estimator inspired by the recently introduced Efficient Manifold Density…
Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers
The Hong Kong University of Science and Technology（香港科学与技术大学）；UNC-Chapel Hill（北卡罗来纳大学教堂山分校）；Microsoft（微软）；The Chinese University of Hong Kong（香港中文大学）；UIUC（伊利诺伊大学厄巴纳-香槟分校）
FACEMUG: A Multimodal Generative and Fusion Framework for Local Facial Editing
College of Computer Science and Artificial Intelligence, Wenzhou University（温州大学计算机与人工智能学院）；Memorial University of Newfoundland（纽芬兰纪念大学）；State Key Laboratory of CAD&CG, Zhejiang University（浙江大学CAD&CG国家重点实验室）
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；University College London（伦敦大学学院）；Chinese University of Hongkong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳市大数据研究院）
The existing text-guided image synthesis methods can only produce limited quality results with at most \mbox{$\text{256}^2$} resolution and the textual instructions are constrained in a small Corpus. In this work, we propose a unified framework for both face image generation and manipulation that produces diverse and high-quality images with an unprecedented resolution at 1024 from multimodal inputs. More importantly, our method supports open-world scenarios, including both image and text, without any re-training, fine-tuning, or post-processing. To be specific, we propose a brand new paradigm…
Think in Strokes, Not Pixels: Process-Driven Image Generation via Interleaved Reasoning
Meta Superintelligence Labs（Meta超级智能实验室）；University of California, San Diego（加州大学圣迭戈分校）；Worcester Polytechnic Institute（伍斯特理工学院）；Northwestern University（西北大学）
Cross-Modal Few-Shot Learning: a Generative Transfer Learning Framework
National Engineering Research Center for Multimedia Software（国家多媒体软件工程技术研究中心）；Institute of Artificial Intelligence（人工智能研究院）；School of Computer Science（计算机学院）；Wuhan University（武汉大学）；Hubei Key Laboratory of Multimedia and Network Communication Engineering（多媒体与网络通信工程湖北省重点实验室）；Centre for Frontier AI Research（前沿人工智能研究中心）；Agency for Science, Technology and Research（新加坡科技研究局）；University of Maryland, College Park（马里兰大学帕克分校）；University of Toronto（多伦多大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）
