MSE-Adapter：一种赋予大型语言模型执行多模态情感分析与情绪识别能力的轻量级插件
MSE-Adapter: A Lightweight Plugin Endowing LLMs with the Capability to Perform Multimodal Sentiment Analysis and Emotion Recognition
AI总结：
针对多模态情感分析和会话情绪识别训练会损害LLM原有能力且计算成本高的问题，本文提出轻量级MSE-Adapter插件，通过TGM融合非文本与文本特征，在四个中英文数据集上验证有效。
摘要：
当前基于预训练语言模型的多模态情感分析（MSA）与会话情绪识别（ERC）方法存在两个主要局限：1）这些预训练语言模型一旦针对MSA和ERC任务完成训练，就会丧失其原有的泛化能力；2）它们需要大量计算资源。随着预训练语言模型规模持续增长，采用以往方法训练更大的多模态情感分析模型可能造成不必要的计算开销。为应对这一挑战，我们提出多模态情感分析与情绪识别适配器（Multimodal Sentiment Analysis and Emotion Recognition Adapter，MSE-Adapter），这是一个轻量且适应性强的插件。该插件使大型语言模型（LLM）能够以极小的计算开销执行MSA或ERC任务（在6/7B模型上仅引入约260万至280万个可训练参数），同时保留LLM的内在能力。在MSE-Adapter中，我们引入Text-Guide-Mixer（TGM）模块，通过Hadamard积在非文本模态与文本模态之间建立显式连接。这使非文本模态能够在特征层面更好地与文本模态对齐，促进生成更高质量的伪标记。我们使用消费级GPU，并以开源LLM（Qwen-1.8B、ChatGLM3-6B-base和LLaMA2-7B）作为骨干模型，在四个公开英文和中文数据集上进行了大量实验。结果证明了所提插件的有效性。代码将在盲审结束后在GitHub上发布。
英文摘要：
Current Multimodal Sentiment Analysis (MSA) and Emotion Recognition in Conversations (ERC) methods based on pre-trained language models exhibit two primary limitations: 1) Once trained for MSA and ERC tasks, these pre-trained language models lose their original generalized capabilities. 2) They demand considerable computational resources. As the size of pre-trained language models continues to grow, training larger multimodal sentiment analysis models using previous approaches could result in unnecessary computational cost. In response to this challenge, we propose \textbf{M}ultimodal \textbf{S}entiment Analysis and \textbf{E}motion Recognition \textbf{Adapter} (MSE-Adapter), a lightweight and adaptable plugin. This plugin enables a large language model (LLM) to carry out MSA or ERC tasks with minimal computational overhead (only introduces approximately 2.6M to 2.8M trainable parameters upon the 6/7B models), while preserving the intrinsic capabilities of the LLM. In the MSE-Adapter, the Text-Guide-Mixer (TGM) module is introduced to establish explicit connections between non-textual and textual modalities through the Hadamard product. This allows non-textual modalities to better align with textual modalities at the feature level, promoting the generation of higher-quality pseudo tokens. Extensive experiments were conducted on four public English and Chinese datasets using consumer-grade GPUs and open-source LLMs (Qwen-1.8B, ChatGLM3-6B-base, and LLaMA2-7B) as the backbone. The results demonstrate the effectiveness of the proposed plugin. The code will be released on GitHub after a blind review.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
