面向视频问答的冗余感知Transformer
Redundancy-aware Transformer for Video Question Answering
AI总结：
本文提出冗余感知Transformer（RaFormer），通过强调邻近帧对象变化、邻近外消息传递和自适应采样区分视觉-语言交互，解决视频问答中的邻近帧冗余与跨模态冗余问题，并在多个基准上取得领先性能。
摘要：
本文指出了当前视频问答（VideoQA）范式中的两类冗余。具体而言，当前的视频编码器倾向于以分层方式在不同粒度上整体嵌入所有视频线索，这不可避免地引入了邻近帧冗余，可能淹没对象层面的细节视觉线索。随后，主流的视觉-语言融合设计通过穷尽地将所有视觉元素与问题token进行融合，而未明确区分它们之间的成对视觉-语言交互，从而引入了跨模态冗余，对回答产生不利影响。为此，我们提出了一种新颖的基于Transformer的架构，旨在以冗余感知的方式对视频问答进行建模。针对邻近帧冗余，我们引入了一种视频编码器结构，强调邻近帧中对象层面的变化，同时采用一种邻近外消息传递方案，仅对远距离帧施加注意力。针对跨模态冗余，我们为融合模块配备了一种新颖的自适应采样机制，通过识别一小部分专门支持答案的视觉元素，明确区分视觉-语言交互。基于这些改进，我们发现该冗余感知Transformer（RaFormer）在多个视频问答基准上取得了最先进的结果。
英文摘要：
This paper identifies two kinds of redundancy in the current VideoQA paradigm. Specifically, the current video encoders tend to holistically embed all video clues at different granularities in a hierarchical manner, which inevitably introduces \textit{neighboring-frame redundancy} that can overwhelm detailed visual clues at the object level. Subsequently, prevailing vision-language fusion designs introduce the \textit{cross-modal redundancy} by exhaustively fusing all visual elements with question tokens without explicitly differentiating their pairwise vision-language interactions, thus making a pernicious impact on the answering. To this end, we propose a novel transformer-based architecture, that aims to model VideoQA in a redundancy-aware manner. To address the neighboring-frame redundancy, we introduce a video encoder structure that emphasizes the object-level change in neighboring frames, while adopting an out-of-neighboring message-passing scheme that imposes attention only on distant frames. As for the cross-modal redundancy, we equip our fusion module with a novel adaptive sampling, which explicitly differentiates the vision-language interactions by identifying a small subset of visual elements that exclusively support the answer. Upon these advancements, we find this \underline{R}edundancy-\underline{a}ware trans\underline{former} (RaFormer) can achieve state-of-the-art results on multiple VideoQA benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
