arXiv 2407.113002024-07-16cs.CVcs.AI

大型视觉-语言模型作为上下文感知中的情绪识别器

Large Vision-Language Models as Emotion Recognizers in Context Awareness

Academy for Engineering and Technology, Fudan University（复旦大学工程与技术学院）；Cognition and Intelligent Technology Laboratory（认知与智能技术实验室）；Engineering Research Center of AI and Robotics（人工智能与机器人工程研究中心）；AI and Unmanned Systems Engineering Research Center of Jilin Province（吉林省人工智能与无人系统工程研究中心）

Yuxuan Lei, Dingkang Yang, Zhaoyu Chen, Jiawei Chen, Peng Zhai, Lihua Zhang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文探索了利用大型视觉-语言模型通过三种范式（微调、零/少样本上下文学习、思维链）提升上下文感知情绪识别性能，实验证明其在少样本下无需大量训练即可达到竞争性效果。

摘要：

上下文感知情绪识别（CAER）是一项复杂且重要的任务，需要从各种上下文线索中感知情绪。以往的方法主要集中于设计复杂的架构来从图像中提取情绪线索。然而，这些方法的知识局限于特定的训练数据集，并可能反映标注者的主观情绪偏见。此外，在现实应用中，获取大量标注数据往往具有挑战性。本文系统地探索了利用大型视觉-语言模型（LVLMs）赋能CAER任务的三种范式：1）我们在两个CAER数据集上微调LVLMs，这是将大型模型迁移到下游任务的最常见方式。2）我们设计了零样本和少样本模式，以评估LVLMs在数据有限甚至完全未见场景下的性能。在这种情况下，我们提出了一种免训练框架，以充分利用LVLMs的上下文学习（ICL）能力。具体而言，我们开发了一种基于图像相似度的排序算法来检索示例；随后，将指令、检索到的示例和测试示例组合输入LVLMs，以获得相应的情感判断。3）为了利用LVLMs丰富的知识库，我们将思维链（CoT）纳入我们的框架，以增强模型的推理能力并提供可解释的结果。大量实验和分析表明，LVLMs在不同范式下在CAER任务中均取得了具有竞争力的性能。值得注意的是，在少样本设置中的优越性能表明，LVLMs无需大量训练即可完成特定任务的可行性。

英文摘要：

Context-aware emotion recognition (CAER) is a complex and significant task that requires perceiving emotions from various contextual cues. Previous approaches primarily focus on designing sophisticated architectures to extract emotional cues from images. However, their knowledge is confined to specific training datasets and may reflect the subjective emotional biases of the annotators. Furthermore, acquiring large amounts of labeled data is often challenging in real-world applications. In this paper, we systematically explore the potential of leveraging Large Vision-Language Models (LVLMs) to empower the CAER task from three paradigms: 1) We fine-tune LVLMs on two CAER datasets, which is the most common way to transfer large models to downstream tasks. 2) We design zero-shot and few-shot patterns to evaluate the performance of LVLMs in scenarios with limited data or even completely unseen. In this case, a training-free framework is proposed to fully exploit the In-Context Learning (ICL) capabilities of LVLMs. Specifically, we develop an image similarity-based ranking algorithm to retrieve examples; subsequently, the instructions, retrieved examples, and the test example are combined to feed LVLMs to obtain the corresponding sentiment judgment. 3) To leverage the rich knowledge base of LVLMs, we incorporate Chain-of-Thought (CoT) into our framework to enhance the model's reasoning ability and provide interpretable results. Extensive experiments and analyses demonstrate that LVLMs achieve competitive performance in the CAER task across different paradigms. Notably, the superior performance in few-shot settings indicates the feasibility of LVLMs for accomplishing specific tasks without extensive training.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。