arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.05640cs.CV
收藏

AffectSeek: 长视频中基于模糊用户查询的代理情感理解

AffectSeek: Agentic Affective Understanding in Long Videos under Vague User Queries

Gansu Provincial Key Laboratory of Wearable Computing, School of Information Science and Engineering, Lanzhou University(甘肃省可穿戴计算重点实验室,兰州大学信息科学与工程学院);Guangdong-Hong Kong-Macao Joint Laboratory for Emotional Intelligence and Pervasive Computing, Shenzhen MSU-BIT University(粤港澳大湾区情感智能与泛在计算联合实验室,深圳MSU-BIT大学);Department of Computer and Information Engineering, Khalifa University(计算机与信息工程系,哈利法大学);Artificial Intelligence Research Institute, Shenzhen MSU-BIT University(人工智能研究院,深圳MSU-BIT大学);Department of Electrical and Computer Engineering, The University of Hong Kong(电子与计算机工程系,香港大学)

Zhen Zhang, Yuhang Yang, Yunxiang Jiang, Yuhuan Lu, Haifeng Lu, Zheng Lian, Runhao Zeng, Xiping Hu

AI总结:

本文提出VQAU任务,要求模型在长视频中定位情感时刻、预测情绪类别并生成证据支持的解释。构建VQAU-Bench基准,并提出AffectSeek框架,通过分步推理实现模糊查询驱动的情感理解。

摘要:

现有情感理解研究主要关注从图像、音频或预剪视频中识别情绪,其中情感证据已给出。这种被动且以片段为中心的设置无法完全反映真实场景,用户常与长视频互动并通过自然语言查询表达需求。本文研究Vague-Query-driven视频情感理解(VQAU)任务,要求模型在长视频中定位情感时刻、预测情绪类别并生成证据支持的解释。为支持该任务,我们构建了VQAU-Bench基准,整合了长视频、模糊情感查询、时间片段标注、情绪标签和解释性说明,形成统一评估框架。VQAU-Bench可系统评估语义-时间-情感对齐、情感时刻定位、情绪分类和解释生成。为解决VQAU的多步推理挑战,我们进一步提出AffectSeek框架,通过主动寻找、验证和解释情感时刻实现长视频情感理解。AffectSeek将VQAU分解为意图解释、候选定位、片段验证、情绪推理和解释生成,并通过角色专业化推理和跨阶段验证逐步对齐模糊用户意图与长视频证据。实验表明,VQAU对现有情感识别模型和单步视觉-语言模型仍具挑战性,而AffectSeek提供了一个简单而有效的代理长视频情感理解框架。

英文摘要:

Existing affective understanding studies have mainly focused on recognizing emotions from images, audio signals, or pre-cliped video clips, where the affective evidence is already given. This passive and clip-centered setting does not fully reflect real-world scenarios, in which users often interact with long videos and express their needs through natural-language queries. In this paper, we study \textbf{Vague-Query-driven video Affective Understanding (VQAU)}, a new task that requires models to localize affective moments in long videos, predict their emotion categories, and generate evidence-grounded rationales under vague user queries. To support this task, we construct \textbf{VQAU-Bench}, a benchmark that integrates long videos, vague affective queries, temporal clip annotations, emotion labels, and rationale explanations into a unified evaluation framework. VQAU-Bench enables systematic assessment of semantic-temporal-affective alignment, affective moment localization, emotion classification, and rationale generation. To address the multi-step reasoning challenges of VQAU, we further propose \textbf{AffectSeek}, an agentic framework that actively seeks, verifies, and explains affective moments in long videos. AffectSeek decomposes VQAU into intent interpretation, candidate localization, clip verification, emotion reasoning, and rationale generation, and progressively aligns vague user intent with long-video evidence through role-specialized reasoning and cross-stage verification. Experiments show that VQAU remains challenging for existing affective recognition models and single-step vision-language models, while AffectSeek provides a simple yet effective framework for agentic long-video affective understanding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑