MASH-VLM：通过解耦时空表征缓解视频大语言模型中的动作-场景幻觉
MASH-VLM: Mitigating Action-Scene Hallucination in Video-LLMs through Disentangled Spatial-Temporal Representations
AI总结：
针对视频大语言模型的动作-场景幻觉问题，提出基于解耦时空表征的MASH-VLM，通过DST-attention与Harmonic-RoPE两项创新，在新建UNSCENE基准及现有视频理解基准上取得最优结果。
摘要：
在本研究中，我们解决了视频大语言模型（Video-LLMs）中的动作-场景幻觉问题，即模型会根据场景上下文错误预测动作，或根据观测到的动作错误推断场景。我们发现现有Video-LLMs易出现动作-场景幻觉主要有两点原因：第一，现有模型通过在所有token上应用注意力操作，导致空间特征与时间特征相互混杂；第二，它们采用标准旋转位置嵌入（RoPE），使得文本token会根据自身序列顺序过度强调特定类型的token。针对这些问题，我们提出MASH-VLM——通过解耦时空表征缓解视频大语言模型中的动作-场景幻觉。该方法包含两项核心创新：（1）DST-attention，一种新型注意力机制，通过掩码注意力限制空间token与时间token的直接交互，在大语言模型内部实现时空token的解耦；（2）Harmonic-RoPE，通过扩展位置ID的维度，使空间与时间token相对于文本token保持均衡的位置。为评估Video-LLMs的动作-场景幻觉问题，我们构建了包含1320个视频、4078个问答对的UNSCENE基准。大量实验表明，MASH-VLM在UNSCENE基准以及现有视频理解基准上均取得了最优结果。
英文摘要：
In this work, we tackle action-scene hallucination in Video Large Language Models (Video-LLMs), where models incorrectly predict actions based on the scene context or scenes based on observed actions. We observe that existing Video-LLMs often suffer from action-scene hallucination due to two main factors. First, existing Video-LLMs intermingle spatial and temporal features by applying an attention operation across all tokens. Second, they use the standard Rotary Position Embedding (RoPE), which causes the text tokens to overemphasize certain types of tokens depending on their sequential orders. To address these issues, we introduce MASH-VLM, Mitigating Action-Scene Hallucination in Video-LLMs through disentangled spatial-temporal representations. Our approach includes two key innovations: (1) DST-attention, a novel attention mechanism that disentangles the spatial and temporal tokens within the LLM by using masked attention to restrict direct interactions between the spatial and temporal tokens; (2) Harmonic-RoPE, which extends the dimensionality of the positional IDs, allowing the spatial and temporal tokens to maintain balanced positions relative to the text tokens. To evaluate the action-scene hallucination in Video-LLMs, we introduce the UNSCENE benchmark with 1,320 videos and 4,078 QA pairs. Extensive experiments demonstrate that MASH-VLM achieves state-of-the-art results on the UNSCENE benchmark, as well as on existing video understanding benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
