用于情感视频描述的双路径协同生成网络
Dual-path Collaborative Generation Network for Emotional Video Captioning
AI总结：
针对情感视频描述中情感动态变化易被忽视及情感引导过强导致事实内容弱化的问题，提出双路径协同生成网络，通过动态情感演变模块和情感自适应解码器实现情感线索动态感知与事实-情感平衡生成，并在三个数据集上验证了有效性。
摘要：
情感视频描述（Emotional Video Captioning）是一项新兴任务，旨在描述视频中的事实内容及其所表达的内在情感。EVC任务的核心在于在描述生成过程中有效感知细微且模糊的视觉情感线索，而传统视频描述方法忽视了这一点。现有的情感视频描述方法首先感知全局视觉情感线索，然后将其与视频特征结合以引导情感描述生成，但忽视了EVC任务的两个特点。首先，这些方法忽视了视频内在情感的动态细微变化，难以满足情感多样且多变的常见场景需求。其次，由于这些方法在每一步都融入情感线索，情感引导作用被过度强调，导致生成过程中事实内容或多或少被忽略。为此，我们提出了一种双路径协同生成网络，通过协同学习在生成情感描述的同时动态感知视觉情感线索的演变。具体而言，在动态情感感知路径中，我们提出了动态情感演变模块，该模块首先聚合视觉特征和历史描述特征以总结全局视觉情感线索，然后动态选择每个阶段需要重新组合的情感线索。此外，在自适应描述生成路径中，为了平衡事实内容描述与情感线索，我们提出了情感自适应解码器。因此，我们的方法能够在必要的时间步生成与情感相关的词语，并且我们的描述生成能够很好地平衡事实内容与情感线索的引导。在三个具有挑战性的数据集上的大量实验证明了我们的方法及所提出的每个模块的优越性。
英文摘要：
Emotional Video Captioning is an emerging task that aims to describe factual content with the intrinsic emotions expressed in videos. The essential of the EVC task is to effectively perceive subtle and ambiguous visual emotional cues during the caption generation, which is neglected by the traditional video captioning. Existing emotional video captioning methods perceive global visual emotional cues at first, and then combine them with the video features to guide the emotional caption generation, which neglects two characteristics of the EVC task. Firstly, their methods neglect the dynamic subtle changes in the intrinsic emotions of the video, which makes it difficult to meet the needs of common scenes with diverse and changeable emotions. Secondly, as their methods incorporate emotional cues into each step, the guidance role of emotion is overemphasized, which makes factual content more or less ignored during generation. To this end, we propose a dual-path collaborative generation network, which dynamically perceives visual emotional cues evolutions while generating emotional captions by collaborative learning. Specifically, in the dynamic emotion perception path, we propose a dynamic emotion evolution module, which first aggregates visual features and historical caption features to summarize the global visual emotional cues, and then dynamically selects emotional cues required to be re-composed at each stage. Besides, in the adaptive caption generation path, to balance the description of factual content and emotional cues, we propose an emotion adaptive decoder. Thus, our methods can generate emotion-related words at the necessary time step, and our caption generation balances the guidance of factual content and emotional cues well. Extensive experiments on three challenging datasets demonstrate the superiority of our approach and each proposed module.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
