面向面部表情识别的多模态提示对齐
Multimodal Prompt Alignment for Facial Expression Recognition
AI总结：
提出多模态提示对齐框架MPA-FER，通过多粒度硬提示生成、原型引导视觉特征对齐和跨模态全局-局部对齐模块，利用LLM外部知识提升细粒度文本-视觉关系捕捉，在三个FER基准数据集上优于现有方法。
摘要：
提示学习已被广泛采用，以高效地将像CLIP这样的视觉-语言模型（VLMs）适应于各种下游任务。尽管取得了成功，当前基于VLM的面部表情识别（FER）方法仍难以捕捉细粒度的文本-视觉关系，而这对于区分面部表情之间的细微差异至关重要。为了应对这一挑战，我们提出了一种用于FER的多模态提示对齐框架，称为MPA-FER，它为提示视觉特征的学习过程提供细粒度的语义指导，从而产生更精确和可解释的表示。具体来说，我们引入了一种多粒度硬提示生成策略，利用像ChatGPT这样的大型语言模型（LLM）为每种面部表情生成详细描述。通过最小化软提示和硬提示之间的特征差异，将基于LLM的外部知识注入到软提示中。为了保留预训练CLIP模型的泛化能力，我们的方法结合了原型引导的视觉特征对齐，确保来自冻结图像编码器的提示视觉特征与特定类别的原型紧密对齐。此外，我们提出了一个跨模态全局-局部对齐模块，专注于与表情相关的面部特征，进一步改善文本和视觉特征之间的对齐。大量实验表明，我们的框架在三个FER基准数据集上优于最先进的方法，同时保留了预训练模型的优势并最小化了计算成本。
英文摘要：
Prompt learning has been widely adopted to efficiently adapt vision-language models (VLMs) like CLIP for various downstream tasks. Despite their success, current VLM-based facial expression recognition (FER) methods struggle to capture fine-grained textual-visual relationships, which are essential for distinguishing subtle differences between facial expressions. To address this challenge, we propose a multimodal prompt alignment framework for FER, called MPA-FER, that provides fine-grained semantic guidance to the learning process of prompted visual features, resulting in more precise and interpretable representations. Specifically, we introduce a multi-granularity hard prompt generation strategy that utilizes a large language model (LLM) like ChatGPT to generate detailed descriptions for each facial expression. The LLM-based external knowledge is injected into the soft prompts by minimizing the feature discrepancy between the soft prompts and the hard prompts. To preserve the generalization abilities of the pretrained CLIP model, our approach incorporates prototype-guided visual feature alignment, ensuring that the prompted visual features from the frozen image encoder align closely with class-specific prototypes. Additionally, we propose a cross-modal global-local alignment module that focuses on expression-relevant facial features, further improving the alignment between textual and visual features. Extensive experiments demonstrate our framework outperforms state-of-the-art methods on three FER benchmark datasets, while retaining the benefits of the pretrained model and minimizing computational costs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
