一个身份，多种角色：多模态实体指代用于增强视频情境识别
One Identity, Many Roles: Multimodal Entity Coreference for Enhanced Video Situation Recognition
AI总结：
本文提出多模态实体指代（MEC）方法，通过结合文本和视频信息提升视频情境识别的准确性和一致性，实验结果显示在描述和视觉定位方面均取得显著提升。
摘要：
视频情境识别（VidSitu）旨在解决视频中'谁对谁做了什么，用什么，如何，在哪里'的复杂问题。它通过识别多个事件中的显著动作及其关联短描述来测试视频理解的深度。通过VidSitu需要在不同镜头和不同外观中对关键实体进行时空定位。我们提出，连贯的视频理解需要一致地识别扮演不同角色的实体。我们提出了多模态实体指代（MEC）来将实体描述与视频中的定位结合在一起。为此，我们引入了CineMEC，一种多阶段方法，将事件角色提及组与实体的视觉聚类结合在一起，无需在训练过程中显式进行定位监督。我们的方法旨在利用视觉定位和描述之间的协同作用，其中改进一个会反过来影响另一个。为了评估，我们扩展了VidSitu数据集，添加了定位注释。虽然先前的工作主要关注描述，CineMEC在描述（+2.5% CIDEr，+7% LEA）和视觉定位（+18% HOTA）方面都提高了一致性。
英文摘要：
Video Situation Recognition (VidSitu) addresses the challenging problem of "who did what to whom, with what, how, and where" in a video. It tests thorough video understanding by requiring identification of salient actions and associated short descriptions for event roles across multiple events. Grounding with VidSitu requires spatio-temporal localization of key entities across shots and varied appearances. We posit that coherent video understanding requires consistent identification of entities that play different roles. We propose Multimodal Entity Coreference (MEC) to unite entity descriptions in text with grounding across the video. Towards this, we introduce CineMEC, a multi-stage approach that unites event role mention groups with visual clusters of entities, without explicit grounding supervision during training. Our approach is designed to exploit the synergy between visual grounding and captioning, where improving one influences the other and vice versa. For evaluation, we extend the VidSitu dataset with grounding annotations. While previous work focuses primarily on descriptions, CineMEC improves consistency across both: captioning (+2.5% CIDEr, +7% LEA) and visual grounding (+18% HOTA).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
