AnomalyLMM: 桥接生成式知识与判别式检索以实现基于文本的人员异常搜索
AnomalyLMM: Bridging Generative Knowledge and Discriminative Retrieval for Text-Based Person Anomaly Search
AI总结：
针对基于文本的人员异常搜索中细粒度跨模态对齐难、异常样本稀疏的问题，提出AnomalyLMM框架，通过免训练的由粗到细流水线桥接生成式知识与判别式检索，在PAB数据集上Recall@1较基线提升0.96%。
摘要：
随着公共安全需求不断增长，基于文本的人员异常搜索已成为一项关键任务，旨在通过自然语言描述检索存在异常行为的人员。与传统人员搜索不同，该任务面临两大独特挑战：（1）文本异常描述与视觉行为之间的细粒度跨模态对齐；（2）真实世界样本稀疏情况下的异常识别。尽管大型多模态模型（Large Multi-modal Models, LMMs）在多模态理解方面表现出色，但其在细粒度异常检索领域的潜力仍未得到充分挖掘，主要受限于两点：（1）生成式知识与判别式检索之间存在领域鸿沟；（2）缺乏适配部署的高效微调策略。\n在本研究中，我们提出AnomalyLMM，这是首个利用LMMs开展基于文本的人员异常搜索的框架。我们的核心贡献包括：（1）一种全新的由粗到细的流水线，整合LMMs以桥接生成式世界知识与以检索为核心的异常检测；（2）一套免训练的适配方案，包含掩码跨模态提示、行为显著性预测和知识感知重排序，能够以零样本方式聚焦于细微的异常线索。\n作为首个探索LMMs应用于该任务的研究，我们在PAB数据集（目前唯一公开的基于文本人员异常搜索基准，其精心整理的真实世界异常覆盖跌倒、碰撞、被击打等多种场景）上开展了严格评估。实验表明所提方法具备有效性，其Recall@1准确率较具竞争力的基线模型提升了+0.96%。值得注意的是，我们的方法展现出文本异常与视觉行为之间可解释的对齐关系，这一点通过定性分析得到验证。我们的代码和模型将公开发布以供未来研究使用。
英文摘要：
With growing public safety demands, text-based person anomaly search has emerged as a critical task, aiming to retrieve individuals with abnormal behaviors via natural language descriptions. Unlike conventional person search, this task presents two unique challenges: (1) fine-grained cross-modal alignment between textual anomalies and visual behaviors, and (2) anomaly recognition under sparse real-world samples. While Large Multi-modal Models (LMMs) excel in multi-modal understanding, their potential for fine-grained anomaly retrieval remains underexplored, hindered by: (1) a domain gap between generative knowledge and discriminative retrieval, and (2) the absence of efficient adaptation strategies for deployment. In this work, we propose AnomalyLMM, the first framework that harnesses LMMs for text-based person anomaly search. Our key contributions are: (1) A novel coarse-to-fine pipeline integrating LMMs to bridge generative world knowledge with retrieval-centric anomaly detection; (2) A training-free adaptation cookbook featuring masked cross-modal prompting, behavioral saliency prediction, and knowledge-aware re-ranking, enabling zero-shot focus on subtle anomaly cues. As the first study to explore LMMs for this task, we conduct a rigorous evaluation on the PAB dataset, the only publicly available benchmark for text-based person anomaly search, with its curated real-world anomalies covering diverse scenarios (e.g., falling, collision, and being hit). Experiments show the effectiveness of the proposed method, surpassing the competitive baseline by +0.96% Recall@1 accuracy. Notably, our method reveals interpretable alignment between textual anomalies and visual behaviors, validated via qualitative analysis. Our code and models will be released for future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
