通过细粒度情感-原因对提取实现精确的情感归因视频字幕生成
Towards Accurate Emotion-Attributed Video Captioning via Fine-grained Emotion-Cause Pair Extraction
AI总结：
提出细粒度情感-原因对提取框架，通过概念感知视觉语义分解和视觉引导情感可解释学习，提升情感视频字幕的准确性和丰富性。
摘要：
情感视频字幕生成（EVC）是一项具有挑战性的任务，旨在为视频生成事实准确且情感丰富的描述。现有的EVC方法利用整体视觉特征挖掘全局情感线索，然后聚合多模态特征以指导情感字幕生成，这忽略了EVC任务的关键特性。视觉情感是由特定的动机原因引发的，这些原因通常只隐含在核心视频片段中。整体挖掘带来了显著的信息冗余和不准确的情感线索。因此，细粒度的视觉原因提取对情感感知和情感归因字幕生成都有促进作用。为此，我们提出了一种用于情感归因视频字幕生成的细粒度情感-原因对提取框架。具体来说，我们通过两轮学习成对的情感和原因特征：1）我们提出了一种概念感知的视觉语义分解模块，通过探索场景、对象和运动概念来增强视觉特征。此外，为了增强情感特征，我们提出了一种视觉引导的情感可解释学习模块，该模块利用视觉时间动态指导情感细化，并通过可靠的VAD向量约束增强可解释的细化过程。2）我们通过在细化前后交叉耦合视觉和情感特征来实现情感-原因对提取，并利用对比损失实现语义强制对齐。总体而言，我们的方法优化了视频的复杂语义理解和情感感知，从而在情感字幕生成中取得了有前景的性能。在三个具有挑战性的数据集上进行的大量实验证明了我们的方法和每个提出模块的优越性，例如，在EVC-MSVD数据集上，BLEU-2和ROUGE-L分别取得了+4.4%和+5.4%的最佳性能。
英文摘要：
Emotional Video Captioning (EVC) is a challenging task that aims to generate factually accurate and emotionally rich descriptions for videos. Existing EVC methods leverage holistic visual features to mine global emotional cues, and then aggregate multimodal features to guide the emotional caption generation, which ignores the critical characteristic of the EVC task. Visual emotions are evoked by specific motivational causes, which are usually only implied in core video segments. The holistic mining brings significant information redundancy and inaccurate emotional cues. Thus, fine-grained visual cause extraction has a facilitative effect on both emotion perception and emotion-attributed caption generation. To this end, we propose a fine-grained emotion-cause pair extraction framework for emotion-attributed video captioning. Specifically, we learn pair-wise emotion and cause features in two rounds: 1) We propose a Concept-aware Visual Semantic Decomposition module to augment visual features by exploring scene, object, and motion concepts. Besides, to enhance emotional features, we propose a Visual-guided Emotion Interpretable Learning module, which guides emotion refinement with visual temporal dynamics, and augments the interpretable refinement process by reliable VAD-vector constraints. 2) We achieve emotion-cause pair extraction by cross-coupling the visual and emotional features before and after refinement, and leverage contrastive loss to achieve semantic forced alignment. Overall, our approach optimizes complex semantic understanding and emotion perception of videos, leading to a promising performance in emotional captioning. Extensive experiments on three challenging datasets demonstrate the superiority of our approach and each proposed module, e.g., achieving the best performances with +4.4% and +5.4% w.r.t. BLEU-2 and ROUGE-L, respectively, on the EVC-MSVD dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
