arXiv 2511.120772025-11-29cs.CV

通过视觉学习听觉：现在是让视觉语言模型理解艺术情感的时候了

Learning to Hear by Seeing: It's Time for Vision Language Models to Understand Artistic Emotion from Sight and Sound

Zhejiang University（浙江大学）；The Hong Kong Polytechnic University（香港理工大学）

Dengming Zhang, Weitao You, Jingxiong Li, Weishen Lin, Wenda Shi, Xue Zhao, Heda Zuo, Junxian Wu, Lingyun Sun

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

VAEmotionLLM通过两阶段框架，利用有限音频预训练使视觉语言模型理解艺术中的多模态情感

摘要：

情感理解对于使大型语言模型（LLMs）更加通用、可靠并符合人类需求至关重要。艺术通过视觉和听觉元素的结合来传达情感，但大多数先前的工作都是以人类为中心的或单模态的，忽视了艺术中故意表达的情感。同时，当前的音频视觉语言模型（AVLMs）通常需要大规模的音频预训练来赋予视觉语言模型（VLMs）听觉能力，这限制了可扩展性。我们提出了视觉锚定的音频视觉情感语言模型（VAEmotionLLM），这是一个两阶段框架，通过有限的音频预训练教给VLM听觉能力，并跨模态理解情感。在第一阶段，视觉引导的音频对齐（VG-Align）通过在同步的音频视频片段上对共享LLM的下一个标记分布进行对齐，将冻结的视觉路径转化为新的音频路径，使在没有大规模音频数据的情况下也能实现听觉。在第二阶段，一个轻量级的跨模态情感适配器（EmoAdapter），由情感增强器和情感监督器组成，注入情感敏感的残差，并应用情感监督以增强跨模态的情感理解。我们还构建了ArtEmoBenchmark，一个以艺术为中心的情感基准，评估在仅音频、仅视觉和音频视觉输入下的内容和情感理解。VAEmotionLLM在ArtEmoBenchmark上实现了最先进的结果，优于仅音频、仅视觉和音频视觉基线。消融实验显示所提出组件是互补的。

英文摘要：

Emotion understanding is critical for making Large Language Models (LLMs) more general, reliable, and aligned with humans. Art conveys emotion through the joint design of visual and auditory elements, yet most prior work is human-centered or single-modality, overlooking the emotion intentionally expressed by the artwork. Meanwhile, current Audio-Visual Language Models (AVLMs) typically require large-scale audio pretraining to endow Visual Language Models (VLMs) with hearing, which limits scalability. We present Vision Anchored Audio-Visual Emotion LLM (VAEmotionLLM), a two-stage framework that teaches a VLM to hear by seeing with limited audio pretraining and to understand emotion across modalities. In Stage 1, Vision-Guided Audio Alignment (VG-Align) distills the frozen visual pathway into a new audio pathway by aligning next-token distributions of the shared LLM on synchronized audio-video clips, enabling hearing without a large audio dataset. In Stage 2, a lightweight Cross-Modal Emotion Adapter (EmoAdapter), composed of the Emotion Enhancer and the Emotion Supervisor, injects emotion-sensitive residuals and applies emotion supervision to enhance cross-modal emotion understanding. We also construct ArtEmoBenchmark, an art-centric emotion benchmark that evaluates content and emotion understanding under audio-only, visual-only, and audio-visual inputs. VAEmotionLLM achieves state-of-the-art results on ArtEmoBenchmark, outperforming audio-only, visual-only, and audio-visual baselines. Ablations show that the proposed components are complementary.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。