通过跨模态注意力可区分性理解视频-语言模型中的时间逻辑一致性
Understanding Temporal Logic Consistency in Video-Language Models through Cross-Modal Attention Discriminability
AI总结：
研究探讨视频-语言模型在时间逻辑一致性问题上的核心原因，提出TCAS方法提升跨模态注意力的时序分辨能力，实验验证了方法对时间逻辑一致性的提升效果。
摘要：
大型语言模型（LLMs）常生成自相矛盾的输出，严重影响其可靠性。在视频-语言模型（Video-LLMs）中，这一现象近期引起研究关注。具体而言，这些模型在基于其 grounding 输出回答重新表述的问题时，无法提供逻辑一致的响应。然而，这一现象的潜在原因仍待探索。本文采用可解释性驱动的方法，分析、统计总结并干预该现象的潜在因素。我们发现，响应不一致的主要原因之一在于跨模态注意力头无法有效区分不同时间戳的视频令牌。为解决这一问题，我们提出了一种注意力增强方法，称为时间条件注意力锐化（TCAS），该方法基于注意力区分构建增强目标，以提升模型的时间分辨率能力，从而改善其时间理解逻辑一致性。实验结果表明，我们的方法显著增强了Video-LLMs的时间逻辑一致性。进一步分析表明，我们的方法确实提高了注意力头的时间可区分性，验证了我们的结论。此外，我们的方法甚至在一般视频时间接地任务中实现了性能提升，表明时间逻辑一致性是时间理解的重要因素。
英文摘要：
Large language models (LLMs) often generate self-contradictory outputs, which severely impacts their reliability and hinders their adoption in practical applications. In video-language models (Video-LLMs), this phenomenon recently draws the attention of researchers. Specifically, these models fail to provide logically consistent responses to rephrased questions based on their grounding outputs. However, the underlying causes of this phenomenon remain underexplored. In this work, we adopt an interpretability-driven approach to analyze, statistically summarize, and intervention the potential factors of the phenomenon. We find that one of the primary reasons for the inconsistency in responses lies in the inability of cross-modal attention heads to effectively distinguish video tokens across different timestamps. To address this, we propose an attention enhancement method called Temporally Conditioned Attention Sharpening (TCAS), which constructs an enhancement objective based on attention distinctions to enhance the model's temporal resolution capability, thereby improving its temporal understanding logic consistency. Experimental results demonstrate that our method significantly enhances the temporal logic consistency of Video-LLMs. Further analyses reveal that our method indeed improves the temporal discriminability of attention heads, validating our conclusions. Additionally, our method even achieves performance improvements in general video temporal grounding tasks, suggesting that temporal logic consistency is an important factor in temporal understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
