基于面部动作单元-文本语义融合的LLM多模态人格识别
LLM-based Multimodal Personality Recognition via Facial Action Unit-Text Semantic Fusion
AI总结：
提出一种基于大语言模型的面部动作单元与文本语义融合框架，通过将AU序列转化为可解释文本描述并与受访者文本回答融合，实现异步视频面试中的人格识别，在AVI-6基准上取得更优性能。
摘要：
由于异步视频面试在现代招聘中的广泛采用，人格识别变得越来越重要。现有方法通常依赖大语言模型分析受访者的文本回答。然而，单模态方法常遭受信息损失（例如忽略面部线索）。相比之下，使用全脸图像或稀疏采样帧的多模态方法可能丢弃对准确人格评估至关重要的细粒度时间动态。为克服这些限制，我们提出一个基于LLM的框架，将面部动作单元与AVI的文本回答进行语义融合。AU序列首先被转换为可解释的文本描述，然后通过LLM与参与者的文本回答融合。一个轻量级回归头将生成的嵌入转换为连续的人格分数，而不破坏底层语义空间。在AVI-6基准上的实验表明，与大多数基线相比，该方法在多个特质上具有更低的预测误差和与人类评分更强的相关性，实现了一致的改进。进一步分析揭示，AU衍生的语义表示为文本回答提供了互补的非语言线索。在LLM内将语义理解与回归预测解耦，也带来了更高的训练稳定性和更清晰的可解释性。总体而言，这些发现表明，AU-文本融合为AVI中的人格识别提供了一个具有心理学基础且计算高效的框架。
英文摘要：
Personality recognition in asynchronous video interviews (AVIs) has become increasingly important due to their widespread adoption in modern recruitment. Existing approaches often rely on large language models (LLMs) to analyze textual responses of interviewees in AVI. However, unimodel methods often suffer from information loss (e.g., ignore facial cues). In contrast, multimodal methods that employ full-face images or sparsely sampled frames can discard fine-grained temporal dynamics critical for accurate personality assessment. To overcome these limitations, we propose an LLM-based framework that semantically fuse facial action units (AUs) with textual responses of AVI. AU sequences are first converted into interpretable textual descriptions, which are then fused with participants' textual responses through an LLM. A lightweight regression head transforms the resulting embeddings into continuous personality scores without disrupting the underlying semantic space. Experiments on the AVI-6 benchmark demonstrate consistent improvements over most baselines, with lower prediction errors and stronger correlations with human-rated scores across multiple traits. Further analysis reveals that AU-derived semantic representations offer complementary non-verbal cues to textual responses. Decoupling semantic understanding from regression prediction within the LLM also leads to greater training stability and clearer interpretability. Overall, these findings demonstrate that AU-text fusion provides a psychologically grounded and computationally efficient framework for personality recognition in AVIs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
