SPECTRUM：通过检索与理解模态进行语义处理和情感感知的视频字幕生成
SPECTRUM: Semantic Processing and Emotion-informed video-Captioning Through Retrieval and Understanding Modalities
AI总结：
提出SPECTRUM框架，通过VTAI和HCOT模块捕捉多模态语义与情感主题，利用检索和双损失优化生成情感可信的视频字幕，在多个数据集上超越现有方法。
摘要：
通过分析视频中的细微细节来捕捉其含义和关键概念，是视频字幕生成中一项基础但具有挑战性的任务。识别视频中的主导情感基调能显著增强对其语境的理解。尽管视频字幕生成受到高度重视，但现有模型往往未能充分处理情感主题，导致字幕生成结果欠佳。为解决这些局限，本文提出了一种新颖的语义处理与情感感知视频字幕生成框架——SPECTRUM（通过检索与理解模态进行语义处理和情感感知的视频字幕生成），旨在生成情感和语义上可信的字幕。凭借我们开创性的结构，SPECTRUM利用视觉文本属性调查（VTAI）来辨别多模态语义和情感主题，并通过整体概念导向主题（HCOT）确定描述性字幕的取向，从而表达具有情感感知和领域熟悉的参考。它们利用视频到文本的检索能力和视频内容的多面性来估计候选字幕的情感概率。然后，通过适当加权嵌入属性向量并应用粗粒度和细粒度的情感概念，确定视频的主导主题，这些概念定义了视频的语境对齐。此外，通过使用两个损失函数，SPECTRUM被优化以整合情感信息并最小化预测误差。在EmVidCap、MSVD和MSRVTT视频字幕数据集上的大量实验表明，我们的模型显著超越了最先进的方法。定量和定性评估突出了该模型准确捕捉和传达视频情感及多模态属性的能力。
英文摘要：
Capturing a video's meaning and critical concepts by analyzing the subtle details is a fundamental yet challenging task in video captioning. Identifying the dominant emotional tone in a video significantly enhances the perception of its context. Despite a strong emphasis on video captioning, existing models often need to adequately address emotional themes, resulting in suboptimal captioning results. To address these limitations, this paper proposes a novel Semantic Processing and Emotion-informed video-Captioning Through Retrieval and Understanding Modalities (SPECTRUM) framework to empower the generation of emotionally and semantically credible captions. Leveraging our pioneering structure, SPECTRUM discerns multimodal semantics and emotional themes using Visual Text Attribute Investigation (VTAI) and determines the orientation of descriptive captions through a Holistic Concept-Oriented Theme (HCOT), expressing emotionally-informed and field-acquainted references. They exploit video-to-text retrieval capabilities and the multifaceted nature of video content to estimate the emotional probabilities of candidate captions. Then, the dominant theme of the video is determined by appropriately weighting embedded attribute vectors and applying coarse- and fine-grained emotional concepts, which define the video's contextual alignment. Furthermore, using two loss functions, SPECTRUM is optimized to integrate emotional information and minimize prediction errors. Extensive experiments on the EmVidCap, MSVD, and MSRVTT video captioning datasets demonstrate that our model significantly surpasses state-of-the-art methods. Quantitative and qualitative evaluations highlight the model's ability to accurately capture and convey video emotions and multimodal attributes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
