arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.14512cs.CV
收藏

视频对话的端到端多模态表示学习

End-to-End Multimodal Representation Learning for Video Dialog

Georgia Institute of Technology(佐治亚理工学院)

Huda Alamri, Anthony Bilic, Michael Hu, Apoorva Beedu, Irfan Essa

摘要:

基于视频的对话任务是一个具有挑战性的多模态学习任务,在过去的几年里受到越来越多的关注,最先进的获得新的性能记录。这一进展主要得益于更强大的基于transformer的语言编码器的适应。尽管取得了这些进展,但现有的方法并不能有效地利用视觉特征来帮助解决任务。最近的研究表明,最先进的模型偏向于文本信息,而不是视觉线索。为了更好地利用可用的视觉信息,这项研究提出了一个新的框架,将3D-CNN网络和基于transformer的网络结合到一个单一的视觉编码器中,以从视频中提取更强大的语义表示。视觉编码器与其他输入形式(如文本和音频)进行端到端的联合训练。AVSD任务的实验表明,在生成和检索任务的基线显着改善。

英文摘要:

Video-based dialog task is a challenging multimodal learning task that has received increasing attention over the past few years with state-of-the-art obtaining new performance records. This progress is largely powered by the adaptation of the more powerful transformer-based language encoders. Despite this progress, existing approaches do not effectively utilize visual features to help solve tasks. Recent studies show that state-of-the-art models are biased toward textual information rather than visual cues. In order to better leverage the available visual information, this study proposes a new framework that combines 3D-CNN network and transformer-based networks into a single visual encoder to extract more robust semantic representations from videos. The visual encoder is jointly trained end-to-end with other input modalities such as text and audio. Experiments on the AVSD task show significant improvement over baselines in both generative and retrieval tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑