EventLens：利用事件感知预训练和跨模态链接增强视觉常识推理
EventLens: Leveraging Event-Aware Pretraining and Cross-modal Linking Enhances Visual Commonsense Reasoning
AI总结：
本文提出EventLens，通过事件感知预训练辅助任务激活LLM对复杂场景的全局理解，并在微调时利用引用标签桥接RoI特征与文本实现细粒度跨模态对齐，从而增强视觉常识推理能力。
摘要：
视觉常识推理（VCR）是一项认知任务，挑战模型回答需要人类常识的视觉问题，并提供解释答案为何正确的理由。随着大型语言模型（LLMs）的出现，探索其在VCR中的适用性是自然且必要的。然而，VCR任务需要更多外部知识来应对其具有挑战性的问题，需要特殊设计来激活LLMs的常识推理能力。此外，大多数现有的多模态LLMs采用对整个输入图像的抽象，这使得难以理解VCR中图像区域与文本之间独特的共指标签，对细粒度对齐构成挑战。为解决这些问题，我们提出EventLens，利用事件感知预训练和跨模态链接来增强VCR。首先，通过模拟人类推理的认知过程，引入事件感知预训练辅助任务，以更好地激活LLM对复杂场景的全局理解。其次，在微调期间，我们进一步利用引用标签将RoI特征与文本桥接，同时保留两种模态的语义。最后，我们使用指令式提示来缩小预训练与微调之间的差距，并使用任务特定的适配器将LLM固有的知识与新的常识更好地整合。实验结果表明了我们提出的辅助任务和细粒度链接策略的有效性。
英文摘要：
Visual Commonsense Reasoning (VCR) is a cognitive task, challenging models to answer visual questions requiring human commonsense, and to provide rationales explaining why the answers are correct. With emergence of Large Language Models (LLMs), it is natural and imperative to explore their applicability to VCR. However, VCR task demands more external knowledge to tackle its challenging questions, necessitating special designs to activate LLMs' commonsense reasoning abilities. Also, most existing Multimodal LLMs adopted an abstraction of entire input image, which makes it difficult to comprehend VCR's unique co-reference tags between image regions and text, posing challenges for fine-grained alignment. To address these issues, we propose EventLens that leverages Event-Aware Pretraining and Cross-modal Linking and EnhanceS VCR. First, by emulating the cognitive process of human reasoning, an Event-Aware Pretraining auxiliary task is introduced to better activate LLM's global comprehension of intricate scenarios. Second, during fine-tuning, we further utilize reference tags to bridge RoI features with texts, while preserving both modality semantics. Finally, we use instruct-style prompts to narrow the gap between pretraining and fine-tuning, and task-specific adapters to better integrate LLM's inherent knowledge with new commonsense. Experimental results show the effectiveness of our proposed auxiliary task and fine-grained linking strategy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
