多句视频字幕生成中的隐式与显式常识
Implicit and Explicit Commonsense for Multi-sentence Video Captioning
AI总结：
提出基于Transformer的视频字幕模型，融合隐式与显式常识知识，在指令生成和传统字幕任务上取得显著提升。
摘要：
现有的密集或段落视频字幕生成方法依赖于视频的整体表示，可能结合学习到的对象/动作表示，来约束分层语言解码器。然而，它们从根本上缺乏推理事件进展、因果关系甚至场景中某些对象功能所需的关于世界的常识知识。为了解决这一局限性，我们提出了一种新颖的基于Transformer的视频字幕生成模型，该模型同时考虑了隐式（视觉-语言和纯语言）和显式（知识库）常识知识。我们展示了这些形式的知识，无论是单独还是组合使用，都能提升生成字幕的质量。此外，受模仿学习启发，我们提出了一项新的指令生成任务，其目标是从视频演示中生成一组语言指令。我们使用AI2-THOR环境生成的ALFRED数据集[54]来形式化该任务。虽然指令生成在概念上类似于段落字幕生成，但它的不同之处在于它表现出更强的对象持久性，以及空间感知和因果句子结构。我们展示了我们的常识知识增强方法在该任务上产生了显著改进（METEOR提升至57%，CIDEr提升至8.5%），并在ActivityNet Captions数据集[29]上取得了更传统视频字幕生成的最新结果。
英文摘要：
Existing dense or paragraph video captioning approaches rely on holistic representations of videos, possibly coupled with learned object/action representations, to condition hierarchical language decoders. However, they fundamentally lack the commonsense knowledge of the world required to reason about progression of events, causality, and even the function of certain objects within a scene. To address this limitation we propose a novel video captioning Transformer-based model, that takes into account both implicit (visuo-lingual and purely linguistic) and explicit (knowledge-base) commonsense knowledge. We show that these forms of knowledge, in isolation and in combination, enhance the quality of produced captions. Further, inspired by imitation learning, we propose a new task of instruction generation, where the goal is to produce a set of linguistic instructions from a video demonstration of its performance. We formalize the task using the ALFRED dataset [54] generated using an AI2-THOR environment. While instruction generation is conceptually similar to paragraph captioning, it differs in the fact that it exhibits stronger object persistence, as well as spatially-aware and causal sentence structure. We show that our commonsense knowledge enhanced approach produces significant improvements on this task (up to 57% in METEOR and 8.5% in CIDEr), as well as the state-of-the-art result on more traditional video captioning in the ActivityNet Captions dataset [29].
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
