StimuVAR：基于多模态大语言模型的时空刺激感知视频情感推理
StimuVAR: Spatiotemporal Stimuli-aware Video Affective Reasoning with Multimodal Large Language Models
AI总结：
StimuVAR提出首个基于多模态大语言模型的以观众为中心的视频情感推理框架，通过帧级与令牌级两级刺激感知机制及VAR指令训练，显著提升模型理解观众情感反应并给出合理解释的能力。
摘要：
预测和推理一段视频会如何影响人类的情感，对于开发具有社会智能的系统至关重要。尽管多模态大语言模型（MLLMs）已展现出令人印象深刻的视频理解能力，但它们往往更关注视频的语义内容，常常忽视情感刺激。因此，现有的大多数MLLMs在估计观众的情感反应并提供合理解释方面存在不足。为解决这一问题，我们提出了StimuVAR，一个面向视频情感推理（VAR）的时空刺激感知框架，并与MLLMs相结合。StimuVAR引入了两级刺激感知机制：帧级感知和令牌级感知。帧级感知涉及对最有可能唤起观众情感的事件视频帧进行采样。令牌级感知在令牌空间中执行管状区域选择，使MLLM专注于由情感触发的时空区域。此外，我们构建了VAR指令数据来进行情感训练，将MLLMs的推理优势引导至情感关注方向，从而增强其情感推理能力。为全面评估VAR的有效性，我们提供了一个包含广泛指标的综合评估协议。StimuVAR是首个基于MLLM的以观众为中心的视频情感推理方法。实验证明了其在理解观众对视频的情感反应以及提供连贯且具有洞察力的解释方面的优越性。我们的代码可在https://github.com/EthanG97/StimuVAR获取。
英文摘要：
Predicting and reasoning how a video would make a human feel is crucial for developing socially intelligent systems. Although Multimodal Large Language Models (MLLMs) have shown impressive video understanding capabilities, they tend to focus more on the semantic content of videos, often overlooking emotional stimuli. Hence, most existing MLLMs fall short in estimating viewers' emotional reactions and providing plausible explanations. To address this issue, we propose StimuVAR, a spatiotemporal Stimuli-aware framework for Video Affective Reasoning (VAR) with MLLMs. StimuVAR incorporates a two-level stimuli-aware mechanism: frame-level awareness and token-level awareness. Frame-level awareness involves sampling video frames with events that are most likely to evoke viewers' emotions. Token-level awareness performs tube selection in the token space to make the MLLM concentrate on emotion-triggered spatiotemporal regions. Furthermore, we create VAR instruction data to perform affective training, steering MLLMs' reasoning strengths towards emotional focus and thereby enhancing their affective reasoning ability. To thoroughly assess the effectiveness of VAR, we provide a comprehensive evaluation protocol with extensive metrics. StimuVAR is the first MLLM-based method for viewer-centered VAR. Experiments demonstrate its superiority in understanding viewers' emotional responses to videos and providing coherent and insightful explanations. Our code is available at https://github.com/EthanG97/StimuVAR
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
