arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2101.12059cs.CVcs.CL
收藏

VX2TEXT:端到端学习基于视频的多模态文本生成

VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs

Columbia University(哥伦比亚大学);Facebook AI(脸书人工智能);Georgia Tech(佐治亚理工学院);Dartmouth(达特茅斯学院)

Xudong Lin, Gedas Bertasius, Jue Wang, Shih-Fu Chang, Devi Parikh, Lorenzo Torresani

摘要:

我们提出了\textsc{Vx2Text},一个框架,从多模态输入的文本生成包括视频加文本,语音,或音频。为了利用Transformer网络,它已被证明是有效的建模语言,每个模态首先转换成一组语言嵌入的可学习的标记器。这使得我们的方法在语言空间中执行多模态融合,从而消除了对ad-hoc跨模态融合模块的需要。为了解决连续输入上标记化的不可微性(例如,视频或音频),我们利用能够进行端到端训练的松弛方案。此外,与以前的仅编码器模型不同,我们的网络包括一个自回归解码器,用于从语言编码器融合的多模态嵌入中生成开放式文本。这使得我们的方法完全生成,并使其直接适用于不同的“视频+ $x$到文本”的问题,而不需要为每个任务设计专门的网络头。该框架不仅概念简单,而且非常有效:实验表明,我们的方法基于一个单一的架构优于国家的最先进的三个基于视频的文本生成任务-字幕,问答和视听场景感知对话。

英文摘要:

We present \textsc{Vx2Text}, a framework for text generation from multimodal inputs consisting of video plus text, speech, or audio. In order to leverage transformer networks, which have been shown to be effective at modeling language, each modality is first converted into a set of language embeddings by a learnable tokenizer. This allows our approach to perform multimodal fusion in the language space, thus eliminating the need for ad-hoc cross-modal fusion modules. To address the non-differentiability of tokenization on continuous inputs (e.g., video or audio), we utilize a relaxation scheme that enables end-to-end training. Furthermore, unlike prior encoder-only models, our network includes an autoregressive decoder to generate open-ended text from the multimodal embeddings fused by the language encoder. This renders our approach fully generative and makes it directly applicable to different "video+$x$ to text" problems without the need to design specialized network heads for each task. The proposed framework is not only conceptually simple but also remarkably effective: experiments demonstrate that our approach based on a single architecture outperforms the state-of-the-art on three video-based text-generation tasks -- captioning, question answering and audio-visual scene-aware dialog.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑