GEM-VPC：一种用于视频段落描述的双图增强多模态融合方法
GEM-VPC: A dual Graph-Enhanced Multimodal integration for Video Paragraph Captioning
AI总结：
本文提出GEM-VPC框架，通过构建视频特定时间图和主题图融合多模态信息与外部知识，并利用节点选择模块提升解码效率，在视频段落描述任务上取得优越性能。
摘要：
视频段落描述（VPC）旨在生成概括视频中关键事件的段落描述。尽管近期取得进展，但仍存在挑战，特别是在有效利用视频固有的多模态信号以及解决词汇长尾分布方面。本文提出了一种新颖的多模态融合描述生成框架，用于VPC，利用来自各种模态和外部知识库的信息。我们的框架构建了两个图：一个是“视频特定”的时间图，捕捉多模态信息与常识知识之间的主要事件和交互；另一个是“主题图”，表示特定主题词汇之间的相关性。这些图作为具有共享编码器-解码器架构的Transformer网络的输入。我们还引入了节点选择模块，通过从图中选择最相关的节点来提高解码效率。我们的结果表明，在基准数据集上表现优越。
英文摘要：
Video Paragraph Captioning (VPC) aims to generate paragraph captions that summarises key events within a video. Despite recent advancements, challenges persist, notably in effectively utilising multimodal signals inherent in videos and addressing the long-tail distribution of words. The paper introduces a novel multimodal integrated caption generation framework for VPC that leverages information from various modalities and external knowledge bases. Our framework constructs two graphs: a 'video-specific' temporal graph capturing major events and interactions between multimodal information and commonsense knowledge, and a 'theme graph' representing correlations between words of a specific theme. These graphs serve as input for a transformer network with a shared encoder-decoder architecture. We also introduce a node selection module to enhance decoding efficiency by selecting the most relevant nodes from the graphs. Our results demonstrate superior performance across benchmark datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
