MSG-BART：用于视频基础对话生成的多粒度场景图增强编码器-解码器语言模型
MSG-BART: Multi-granularity Scene Graph-Enhanced Encoder-Decoder Language Model for Video-grounded Dialogue Generation
AI总结：
提出MSG-BART方法，通过将多粒度时空场景图整合进编码器-解码器预训练语言模型，并引入多指针网络促进文本与视频间的信息选择，显著提升了视频基础对话生成的性能。
摘要：
生成基于视频的对话需要对视频中的视觉场景具有高度的理解和推理能力。然而，现有的大型视觉语言模型由于其潜在特征和仅解码器结构而效果不佳，特别是在时空关系推理方面。在本文中，我们提出了一种名为MSG-BART的新方法，通过将多粒度时空场景图纳入编码器-解码器预训练语言模型来增强视频信息的整合。具体而言，我们分别将全局和局部场景图整合到编码器和解码器中，以提升整体感知和目标推理能力。为进一步提升信息选择能力，我们提出了一个多指针网络，以促进文本和视频之间的选择。在三个视频基础对话基准上进行了大量实验，结果表明所提出的MSG-BART相比一系列最先进的方法具有显著的优越性。
英文摘要：
Generating dialogue grounded in videos requires a high level of understanding and reasoning about the visual scenes in the videos. However, existing large visual-language models are not effective due to their latent features and decoder-only structure, especially with respect to spatio-temporal relationship reasoning. In this paper, we propose a novel approach named MSG-BART, which enhances the integration of video information by incorporating a multi-granularity spatio-temporal scene graph into an encoder-decoder pre-trained language model. Specifically, we integrate the global and local scene graph into the encoder and decoder, respectively, to improve both overall perception and target reasoning capability. To further improve the information selection capability, we propose a multi-pointer network to facilitate selection between text and video. Extensive experiments are conducted on three video-grounded dialogue benchmarks, which show the significant superiority of the proposed MSG-BART compared to a range of state-of-the-art approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
