弥合法医图像检索中的模态差距
Bridging the Modality Gap in Forensic Image Retrieval
AI总结：
提出统一检索框架，利用多模态大语言模型生成文本描述并结合视觉与文本特征融合，提升纹身、人脸素描等法医任务的检索精度与鲁棒性。
摘要：
自动图像检索在现代法医分析中扮演着越来越关键的角色，支持依赖于视觉证据高效比较的调查工作流程。虽然先前的工作主要集中在开发和优化多模态检索系统，但很少关注评估这些技术在多样化真实场景中的法医适用性。在本研究中，我们提出了一个统一的检索框架，适用于四个关键的法医任务：（1）给定纹身查询图像的纹身图像检索；（2）由人类专家文本描述引导的纹身检索，模拟目击者口头描述纹身的常见情况；（3）从手绘草图中检索纹身；（4）从法医面部素描中检索人脸。我们的系统利用多模态大语言模型（MLLM）自动为所有查询和图库图像生成结构化文本描述，然后使用句子变换器嵌入进行基于文本的比较。我们使用仅视觉嵌入、仅文本嵌入以及一种多模态融合策略来评估检索性能，该策略结合了来自与每个任务相关的最先进视觉特征提取器的文本和图像相似性分数。模态融合一致地提高了检索精度和鲁棒性，特别是在视觉信息有限或嘈杂的场景中（例如，素描、部分纹身或零碎的目击者陈述）。这项工作突显了统一多模态检索流程的法医价值，并展示了现代MLLM如何能够操作化传统上依赖人工专家分析的具有挑战性的法医任务。我们的结果将多模态检索定位为支持涉及纹身、面部合成和目击者描述的调查工作流程的有前途工具。
英文摘要：
Automated image retrieval plays an increasingly critical role in modern forensic analysis, supporting investigative workflows that rely on efficient comparison of visual evidence. While prior work has focused primarily on developing and optimizing multimodal retrieval systems, limited attention has been paid to evaluating the forensic applicability of these technologies across diverse real-world scenarios. In this study, we present a unified retrieval framework adapted to four key forensic tasks: (1) tattoo image retrieval given a tattoo query image; (2) tattoo retrieval guided by human-expert textual descriptions, modelling the common situation where a witness verbally describes a tattoo; (3) tattoo retrieval from hand-drawn sketches; and (4) face retrieval from forensic face sketches. Our system leverages a multimodal large language model (MLLM) to automatically generate structured textual descriptions for all queries and gallery images, followed by sentence-transformer embedding for text-based comparison. We evaluate retrieval using visual-only embeddings, text-only embeddings and a multimodal fusion strategy that combines text- and image-based similarity scores derived from state-of-the-art visual feature extractors relevant to each task. The fusion of modalities consistently improves retrieval precision and robustness, especially in scenarios where visual information is limited or noisy (e.g., sketches, partial tattoos, or fragmented witness statements). This work highlights the forensic value of a unified multimodal retrieval pipeline and demonstrates how modern MLLMs can operationalize challenging forensic tasks that traditionally rely on manual expert analysis. Our results position multimodal retrieval as a promising tool for supporting investigative workflows involving tattoos, facial composites, and witness descriptions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
