从语义、场景到实例感知：为接地开放词汇情境识别蒸馏基础模型
From Semantics, Scene to Instance-awareness: Distilling Foundation Model for Grounded Open-vocabulary Situation Recognition
AI总结：
针对传统GSR模型泛化能力不足、MLLMs资源消耗大的问题，本文提出Ov-GSR任务与MIPD框架，通过从MLLM蒸馏知识增强小型模型性能，在Ov-SWiG和HICO-DET数据集上取得优异结果。
摘要：
近年来，多模态大语言模型（MLLMs）展现出强大的零样本能力，但在复杂的接地情境识别（GSR）任务中表现不佳，且在边缘设备部署时资源消耗大。同时，传统GSR模型往往缺乏泛化能力，在识别未见和罕见情境时表现不足。本文中，我们利用从教师MLLM向小型GSR模型迁移知识的方法，以增强其泛化能力和零样本能力，从而提出开放词汇接地情境识别（Ov-GSR）任务。为实现这一目标，我们提出多模态交互式提示蒸馏（MIPD），这是一种从基础模型中蒸馏丰富多模态知识的新型框架，使学生Ov-GSR模型能够识别未见情境并更好地感知罕见情境。具体而言，MIPD框架首先利用基于LLM的判断性理据生成器（JRG）构建富含上下文语义信息的正负瞥见和注视理据。然后引入所提出的场景感知和实例感知提示，通过负引导多模态提示对齐（NMPA）模块将理据与来自教师MLLM的视觉信息对齐，有效捕捉整体和感知多模态知识。最后，将对齐后的多模态知识蒸馏到学生Ov-GSR模型中，为泛化提供更坚实的基础，从而增强情境理解、弥合可见与未见场景之间的差距，并减轻罕见案例中的预测偏差。我们在精炼的Ov-SWiG数据集上评估MIPD，在可见、罕见和未见情境中均取得了优异性能，并在HICO-DET数据集上进一步展示了改进的未见检测效果。
英文摘要：
Recent Multimodal Large Language Models (MLLMs) exhibit strong zero-shot abilities but struggle with complex Grounded Situation Recognition (GSR) and are resource-intensive for edge device deployment. Meanwhile, conventional GSR models often lack generalization ability, falling short in recognizing unseen and rare situations. In this paper, we exploit transferring knowledge from a teacher MLLM to a small GSR model to enhance its generalization and zero-shot abilities, thereby introducing the task of Open-vocabulary Grounded Situation Recognition (Ov-GSR). To achieve this, we propose Multimodal Interactive Prompt Distillation (MIPD), a novel framework that distills enriched multimodal knowledge from the foundation model, enabling the student Ov-GSR model to recognize unseen situations and be better aware of rare situations. Specifically, the MIPD framework first leverages the LLM-based Judgmental Rationales Generator (JRG) to construct positive and negative glimpse and gaze rationales enriched with contextual semantic information. The proposed scene-aware and instance-perception prompts are then introduced to align rationales with visual information from the MLLM teacher via the Negative-Guided Multimodal Prompting Alignment (NMPA) module, effectively capturing holistic and perceptual multimodal knowledge. Finally, the aligned multimodal knowledge is distilled into the student Ov-GSR model, providing a stronger foundation for generalization that enhances situation understanding, bridges the gap between seen and unseen scenarios, and mitigates prediction bias in rare cases. We evaluate MIPD on the refined Ov-SWiG dataset, achieving superior performance on seen, rare, and unseen situations, and further demonstrate improved unseen detection on the HICO-DET dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
