超越类别的视觉：基于语言解释器的零样本接地情境识别
Seeing Beyond Classes: Zero-Shot Grounded Situation Recognition via Language Explainer
AI总结：
本文提出基于语言解释器（LEX）的零样本接地情境识别方法，通过动词、接地和名词三个辅助解释器增强模型对动词/名词类别的理解，解决动词概念模糊、角色定位不准和名词预测缺乏上下文感知的问题，在SWiG数据集上验证了其有效性。
摘要：
得益于强大的泛化能力，预训练视觉语言模型（VLMs），例如CLIP，已被广泛应用于零样本场景理解。与简单的识别任务不同，接地情境识别（GSR）要求模型不仅要对图像中的显著活动（动词）进行分类，还要检测参与该动作的所有语义角色。这项复杂任务通常包含三个步骤：动词识别、语义角色接地和名词识别。直接将基于类别的提示与VLMs和接地模型用于此任务存在几个局限性，例如，它难以区分模糊的动词概念，难以在输入固定以动词为中心的模板时准确地对角色进行定位，以及难以实现上下文感知的名词预测。在本文中，我们认为这些局限性源于模型对动词/名词类别的理解不足。为此，我们引入了一种通过语言解释器（LEX）进行零样本GSR的新方法，该方法通过三个解释器显著提升了模型的综合能力：1）动词解释器，生成通用的以动词为中心的描述，以增强不同动词类别的可区分性；2）接地解释器，重述以动词为中心的模板以便更清晰地理解，从而增强精确的语义角色定位；3）名词解释器，创建特定场景的名词描述以确保上下文感知的名词识别。通过在GSR过程的每个步骤中配备辅助解释器，LEX促进了在真实场景中的复杂场景理解。我们在SWiG数据集上的广泛验证证明了LEX在零样本GSR中的有效性和互操作性。
英文摘要：
Benefiting from strong generalization ability, pre-trained vision language models (VLMs), e.g., CLIP, have been widely utilized in zero-shot scene understanding. Unlike simple recognition tasks, grounded situation recognition (GSR) requires the model not only to classify salient activity (verb) in the image, but also to detect all semantic roles that participate in the action. This complex task usually involves three steps: verb recognition, semantic role grounding, and noun recognition. Directly employing class-based prompts with VLMs and grounding models for this task suffers from several limitations, e.g., it struggles to distinguish ambiguous verb concepts, accurately localize roles with fixed verb-centric template1 input, and achieve context-aware noun predictions. In this paper, we argue that these limitations stem from the mode's poor understanding of verb/noun classes. To this end, we introduce a new approach for zero-shot GSR via Language EXplainer (LEX), which significantly boosts the model's comprehensive capabilities through three explainers: 1) verb explainer, which generates general verb-centric descriptions to enhance the discriminability of different verb classes; 2) grounding explainer, which rephrases verb-centric templates for clearer understanding, thereby enhancing precise semantic role localization; and 3) noun explainer, which creates scene-specific noun descriptions to ensure context-aware noun recognition. By equipping each step of the GSR process with an auxiliary explainer, LEX facilitates complex scene understanding in real-world scenarios. Our extensive validations on the SWiG dataset demonstrate LEX's effectiveness and interoperability in zero-shot GSR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
