VISAFF: 以说话者为中心的视觉情感特征学习用于对话中的情感识别
VISAFF: Speaker-Centered Visual Affective Feature Learning for Emotion Recognition in Conversation
AI总结：
本文提出VISAFF框架，通过以说话者为中心的视觉情感特征学习方法，解决对话中情感识别中的复杂场景问题，提升计算效率并避免大规模模型微调的高成本。
摘要：
对话中情感识别（ERC）对于有效的人机交互至关重要，旨在识别多轮对话中说话者的情感状态。早期基于文本的方法在处理如讽刺等复杂场景时存在困难，因为它们本质上忽略了关键的非语言信息。尽管最近的视觉-语言模型（VLMs）通过直接分析视频来解决这一问题，但它们并非专门为ERC量身定制，通常关注与情感无关的背景区域或被动听众，而非活跃说话者。此外，微调这些大模型会带来高昂的计算成本。此外，孤立的视觉信号在缺乏语言内容和语音语调的上下文时往往模糊或技术上受损。为了解决这些挑战，我们提出了VISAFF，一个以说话者为中心的视觉情感特征学习框架用于ERC。VISAFF包括两个阶段：说话者中心的情感定位和可靠性引导的情感补充。VISAFF采用无微调的方法来解锁冻结的VLMs的推理能力，高效地引导它们专注于活跃说话者的情感视觉线索，而无需沉重的训练开销。在第二阶段，我们引入了可靠性引导的情感补充机制，动态利用文本和声音模态来补偿视觉不确定性。在两个真实世界数据集上的实验表明，VISAFF在无微调设置下实现了与最先进方法相媲美的性能，显著提高了计算效率，通过消除对大规模VLMs昂贵微调的需要。源代码可在https://anonymous.4open.science/r/speaker-2365/上获得。
英文摘要：
Emotion Recognition in Conversation (ERC) is essential for effective human-machine interaction, aiming to identify speakers' emotional states in multi-turn dialogues. Early text-based methods struggle with complex scenarios like sarcasm because they inherently neglect vital non-verbal information. While recent Vision-Language Models (VLMs) address this by analyzing video directly, they are not inherently tailored for ERC and often focus on emotionally irrelevant background regions or passive listeners rather than the active speaker. Furthermore, fine-tuning these large models incurs prohibitive computational costs. Additionally, isolated visual signals are frequently ambiguous or technically compromised without the context of linguistic content and vocal prosody. To address these challenges, we propose VISAFF, a speaker-centered VISual AFFective feature learning framework for ERC. VISAFF consists of two stages: Speaker-Centered Affective Grounding and Reliability-Guided Affective Complementation. VISAFF utilizes a tuning-free approach to unlock the reasoning capabilities of frozen VLMs, efficiently steering them to focus on the active speaker's emotional visual cues without heavy training overheads. In the second stage, we introduce a reliability-guided affective complementation mechanism that dynamically leverages textual and acoustic modalities to compensate for visual uncertainty. Experiments on two real-world datasets demonstrate that VISAFF achieves highly competitive performance compared to state-of-the-art methods in a tuning-free setting, significantly enhancing computational efficiency by eliminating the need for expensive fine-tuning of large VLMs. The source code is available at https://anonymous.4open.science/r/speaker-2365/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
