EmoVIT：以视觉指令微调革新情感洞察
EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuning
AI总结：
本文提出EmoVIT架构，通过GPT辅助生成情感视觉指令数据并结合大型语言模型，有效解决了视觉情感理解中指令数据稀缺的问题，在情感分类、情感推理和幽默理解任务上展现出卓越性能。
摘要：
视觉指令微调代表了一种新颖的学习范式，涉及使用特定任务指令对预训练语言模型进行微调。该范式在各种自然语言处理任务中展现出有前景的零样本结果，但在视觉情感理解方面仍尚未被探索。在本工作中，我们专注于增强模型在理解和遵循与情感上下文相关指令方面的能力。最初，我们识别对视觉情感识别至关重要的关键视觉线索。随后，我们引入一种新颖的GPT辅助流水线，用于生成情感视觉指令数据，有效解决该领域标注指令数据稀缺的问题。在InstructBLIP建立的基础工作之上，我们提出的EmoVIT架构结合了情感特定指令数据，利用大型语言模型的强大能力来提升性能。通过广泛实验，我们的模型展示了其在情感分类方面的熟练度、在情感推理方面的技巧以及在理解幽默方面的能力。比较分析为LLMs时代的情感视觉指令微调提供了稳健的基准，提供了有价值的见解并为该领域的未来探索开辟了途径。我们的代码可在https://github.com/aimmemotion/EmoVIT获取。
英文摘要：
Visual Instruction Tuning represents a novel learning paradigm involving the fine-tuning of pre-trained language models using task-specific instructions. This paradigm shows promising zero-shot results in various natural language processing tasks but is still unexplored in vision emotion understanding. In this work, we focus on enhancing the model's proficiency in understanding and adhering to instructions related to emotional contexts. Initially, we identify key visual clues critical to visual emotion recognition. Subsequently, we introduce a novel GPT-assisted pipeline for generating emotion visual instruction data, effectively addressing the scarcity of annotated instruction data in this domain. Expanding on the groundwork established by InstructBLIP, our proposed EmoVIT architecture incorporates emotion-specific instruction data, leveraging the powerful capabilities of Large Language Models to enhance performance. Through extensive experiments, our model showcases its proficiency in emotion classification, adeptness in affective reasoning, and competence in comprehending humor. The comparative analysis provides a robust benchmark for Emotion Visual Instruction Tuning in the era of LLMs, providing valuable insights and opening avenues for future exploration in this domain. Our code is available at \url{https://github.com/aimmemotion/EmoVIT}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
