用于视频字幕生成的协同三流Transformer
Collaborative Three-Stream Transformers for Video Captioning
AI总结：
本文提出协同三流Transformer（COST）框架，分别建模主语、谓语和宾语，并通过跨粒度注意力对齐三个分支的视觉-语言交互，在三个视频字幕数据集上取得了优于现有方法的性能。
摘要：
作为句子中最关键的组成部分，主语、谓语和宾语在视频字幕生成任务中需要特别关注。为了实现这一思想，我们设计了一个新颖的框架，名为协同三流Transformer（COST），分别对这三部分进行建模，并使其相互补充以获得更好的表示。具体来说，COST由三个Transformer分支组成，分别利用视频与文本、检测到的物体与文本、动作与文本之间在时空域中不同粒度的视觉-语言交互。同时，我们提出了一个跨粒度注意力模块来对齐三个Transformer分支所建模的交互，从而使三个Transformer分支能够相互支持，挖掘不同粒度中最具判别力的语义信息，以实现对字幕的准确预测。整个模型以端到端方式进行训练。在三个大规模具有挑战性的数据集（即YouCookII、ActivityNet Captions和MSVD）上进行的大量实验表明，所提出的方法相比当前最先进的方法具有优越的性能。
英文摘要：
As the most critical components in a sentence, subject, predicate and object require special attention in the video captioning task. To implement this idea, we design a novel framework, named COllaborative three-Stream Transformers (COST), to model the three parts separately and complement each other for better representation. Specifically, COST is formed by three branches of transformers to exploit the visual-linguistic interactions of different granularities in spatial-temporal domain between videos and text, detected objects and text, and actions and text. Meanwhile, we propose a cross-granularity attention module to align the interactions modeled by the three branches of transformers, then the three branches of transformers can support each other to exploit the most discriminative semantic information of different granularities for accurate predictions of captions. The whole model is trained in an end-to-end fashion. Extensive experiments conducted on three large-scale challenging datasets, i.e., YouCookII, ActivityNet Captions and MSVD, demonstrate that the proposed method performs favorably against the state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
