Tem-adapter：为视频问答适配图像-文本预训练
Tem-adapter: Adapting Image-Text Pretraining for Video Question Answer
AI总结：
针对视频模型训练成本高及图像-视频域差距问题，本文提出 Tem-Adapter，通过时序对齐器和语义对齐器迁移图文预训练知识，并在两个 VideoQA 基准上验证其显著性能提升。
摘要：
视频-语言预训练模型在指导视频问答（VideoQA）任务方面已展现出显著成功。然而，由于视频序列较长，训练大规模基于视频的模型所产生的成本远高于训练基于图像的模型。这促使我们利用基于图像的预训练中的知识，尽管图像与视频领域之间存在明显差距。为弥合这些差距，本文提出 Tem-Adapter，它通过视觉时序对齐器和文本语义对齐器来学习时序动态与复杂语义。不同于传统预训练知识适配方法仅关注下游任务目标，Temporal Aligner 引入了一个额外的语言引导自回归任务，旨在促进时序依赖关系的学习，其目标是依据历史线索以及描述事件进展的语言引导来预测未来状态。此外，为缩小语义差距并调整文本表示以更好地描述事件，我们引入 Semantic Aligner：它首先设计模板，将问答对融合为事件描述，随后以整个视频序列为引导学习一个 Transformer 解码器进行细化。我们在两个 VideoQA 基准上评估了 Tem-Adapter 与不同的预训练迁移方法，显著的性能提升证明了本文方法的有效性。
英文摘要：
Video-language pre-trained models have shown remarkable success in guiding video question-answering (VideoQA) tasks. However, due to the length of video sequences, training large-scale video-based models incurs considerably higher costs than training image-based ones. This motivates us to leverage the knowledge from image-based pretraining, despite the obvious gaps between image and video domains. To bridge these gaps, in this paper, we propose Tem-Adapter, which enables the learning of temporal dynamics and complex semantics by a visual Temporal Aligner and a textual Semantic Aligner. Unlike conventional pretrained knowledge adaptation methods that only concentrate on the downstream task objective, the Temporal Aligner introduces an extra language-guided autoregressive task aimed at facilitating the learning of temporal dependencies, with the objective of predicting future states based on historical clues and language guidance that describes event progression. Besides, to reduce the semantic gap and adapt the textual representation for better event description, we introduce a Semantic Aligner that first designs a template to fuse question and answer pairs as event descriptions and then learns a Transformer decoder with the whole video sequence as guidance for refinement. We evaluate Tem-Adapter and different pre-train transferring methods on two VideoQA benchmarks, and the significant performance improvement demonstrates the effectiveness of our method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
