arXiv 2503.169452025-03-21cs.CV

PE-CLIP: 面向动态面部表情识别的视觉语言模型参数高效微调方法

PE-CLIP: A Parameter-Efficient Fine-Tuning of Vision Language Models for Dynamic Facial Expression Recognition

Univ. BTU Cottbus-Senftenberg（科特布斯-森夫滕堡应用科学大学）；Sorbonne University Abu Dhabi（索邦大学阿布扎比分校）；Univ. Polytechnique Hauts-de-France（上法兰西理工大学）；Centrale Lille（里尔中央理工学院）；CNRS（法国国家科学研究中心）

Ibtissam Saadi, Abdenour Hadid, Douglas W. Cunningham, Abdelmalik Taleb-Ahmed, Yassin El Hillali

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对动态面部表情识别中视觉语言模型微调效率低、时序建模弱的问题，提出带专用适配器与多模态提示学习的PE-CLIP参数高效微调框架，在两个基准上以更少参数取得媲美最优方法的性能。

摘要：

CLIP等视觉语言模型（VLM）为动态面部表情识别（DFER）提供了有前景的解决方案，但面临全参数微调效率低、复杂度高、文本与视觉表征对齐性差的挑战，且现有方法存在时序建模效果不佳的问题。为解决上述问题，本文提出PE-CLIP，这是一种适配CLIP用于DFER的参数高效微调（PEFT）框架，可在保持高精度的同时大幅减少可训练参数。PE-CLIP引入两个专用适配器：时序动态适配器（TDA）和共享适配器（ShA）。TDA是基于GRU、带动态缩放的模块，可捕获序列依赖关系，同时突出信息丰富的时序特征、抑制无关变化。ShA是轻量适配器，可优化文本和视觉编码器内的表征，保证一致性与效率。此外，本文集成多模态提示学习（MaPLe），为视觉输入和基于动作单元的文本输入引入可学习提示，增强模态间语义对齐，实现CLIP针对动态任务的高效适配。本文在DFEW和FERV39K两个基准数据集上评估PE-CLIP，其性能与当前最优方法相当，且所需可训练参数更少。通过平衡效率与精度，PE-CLIP为资源高效型DFER设立了新基准。所提PE-CLIP的源代码将在https://github.com/Ibtissam-SAADI/PE-CLIP 公开。

英文摘要：

Vision-Language Models (VLMs) like CLIP offer promising solutions for Dynamic Facial Expression Recognition (DFER) but face challenges such as inefficient full fine-tuning, high complexity, and poor alignment between textual and visual representations. Additionally, existing methods struggle with ineffective temporal modeling. To address these issues, we propose PE-CLIP, a parameter-efficient fine-tuning (PEFT) framework that adapts CLIP for DFER while significantly reducing trainable parameters while maintaining high accuracy. PE-CLIP introduces two specialized adapters: a Temporal Dynamic Adapter (TDA) and a Shared Adapter (ShA). The TDA is a GRU-based module with dynamic scaling that captures sequential dependencies while emphasizing informative temporal features and suppressing irrelevant variations. The ShA is a lightweight adapter that refines representations within both textual and visual encoders, ensuring consistency and efficiency. Additionally, we integrate Multi-modal Prompt Learning (MaPLe), introducing learnable prompts for visual and action unit-based textual inputs, enhancing semantic alignment between modalities and enabling efficient CLIP adaptation for dynamic tasks. We evaluate PE-CLIP on two benchmark datasets, DFEW and FERV39K, achieving competitive performance compared to state-of-the-art methods while requiring fewer trainable parameters. By balancing efficiency and accuracy, PE-CLIP sets a new benchmark in resource-efficient DFER. The source code of the proposed PE-CLIP will be publicly available at https://github.com/Ibtissam-SAADI/PE-CLIP .

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。