使用基于图的句子摘要的密集视频字幕生成
Dense Video Captioning using Graph-based Sentence Summarization
AI总结：
提出基于图的分区与摘要（GPaS）两阶段框架，通过GCN-LSTM交互模块融合语义词图与视觉线索进行句子摘要，解决长视频提案中场景演化问题，在ActivityNet Captions和YouCook II上表现优异。
摘要：
最近，密集视频字幕生成在检测和为长未剪辑视频中的所有事件生成字幕方面取得了引人注目的进展。尽管取得了有希望的结果，但大多数现有方法没有充分探索事件时间提案内的场景演化以进行字幕生成，因此当场景和对象在相对较长的提案内发生变化时，表现不太令人满意。为了解决这个问题，我们提出了一个基于图的分区与摘要（GPaS）框架，用于两阶段内的密集视频字幕生成。对于“分区”阶段，将整个事件提案分割成短视频片段，以在更细的层次上生成字幕。对于“摘要”阶段，将每个片段携带丰富描述信息的生成句子总结成一个句子来描述整个事件。我们特别关注“摘要”阶段，并提出了一个有效利用语义词之间关系进行摘要的框架。我们通过将语义词视为图中的节点，并在视觉线索的辅助下，通过耦合图卷积网络（GCN）和长短期记忆网络（LSTM）来学习它们的交互，从而实现这一目标。提出了两种图卷积网络-长短期记忆交互（GLI）模块方案，用于无缝集成GCN和LSTM。通过在两个基准ActivityNet Captions数据集和YouCook II数据集上与最先进方法进行广泛比较，证明了我们方法的有效性。
英文摘要：
Recently, dense video captioning has made attractive progress in detecting and captioning all events in a long untrimmed video. Despite promising results were achieved, most existing methods do not sufficiently explore the scene evolution within an event temporal proposal for captioning, and therefore perform less satisfactorily when the scenes and objects change over a relatively long proposal. To address this problem, we propose a graph-based partition-and-summarization (GPaS) framework for dense video captioning within two stages. For the ``partition" stage, a whole event proposal is split into short video segments for captioning at a finer level. For the ``summarization" stage, the generated sentences carrying rich description information for each segment are summarized into one sentence to describe the whole event. We particularly focus on the ``summarization" stage, and propose a framework that effectively exploits the relationship between semantic words for summarization. We achieve this goal by treating semantic words as nodes in a graph and learning their interactions by coupling Graph Convolutional Network (GCN) and Long Short Term Memory (LSTM), with the aid of visual cues. Two schemes of GCN-LSTM Interaction (GLI) modules are proposed for seamless integration of GCN and LSTM. The effectiveness of our approach is demonstrated via an extensive comparison with the state-of-the-arts methods on the two benchmarks ActivityNet Captions dataset and YouCook II dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
