评估LLM在真实世界面试数据中对人格推断的对齐度
Evaluating LLM Alignment on Personality Inference from Real-World Interview Data
AI总结：
本研究提出包含半结构化面试记录和连续Big Five特质得分的新基准，评估LLM在真实对话中的人格推断对齐度，发现模型预测与真实特质的相关性低于0.26，表明当前LLM对心理学构念的对齐有限。
摘要：
大型语言模型（LLMs）正日益被部署在需要细致心理学理解的角色中，如情感支持智能体、心理咨询师和决策助手。然而，它们解读人类人格特质的能力——此类应用的一个关键方面——仍未被探索，尤其是在具有生态效度的对话环境中。虽然先前的工作利用社交媒体数据上的离散Big Five标签模拟了LLM“人格”，但LLM与源自自然互动的连续真实人格评估之间的对齐度在很大程度上尚未得到检验。为了填补这一空白，我们引入了一个新颖的基准，包含半结构化面试记录以及经过验证的连续Big Five特质得分。使用该数据集，我们系统评估了三种范式下的LLM性能：（1）使用GPT-4.1 Mini的零样本和思维链提示；（2）应用于RoBERTa和Meta-LLaMA架构的基于LoRA的微调；（3）使用预训练BERT和OpenAI的text-embedding-3-small的静态嵌入进行回归。我们的结果显示，模型预测与真实人格特质之间的所有Pearson相关系数均低于0.26，突显了当前LLM与已验证心理学构念之间有限的对齐度。思维链提示相比零样本仅有微小提升，表明人格推断更多依赖于潜在语义表征而非显式推理。这些发现强调了LLM与复杂人类属性对齐的挑战，并推动了未来在特质特定提示、上下文感知建模和面向对齐的微调方面的工作。
英文摘要：
Large Language Models (LLMs) are increasingly deployed in roles requiring nuanced psychological understanding, such as emotional support agents, counselors, and decision-making assistants. However, their ability to interpret human personality traits, a critical aspect of such applications, remains unexplored, particularly in ecologically valid conversational settings. While prior work has simulated LLM "personas" using discrete Big Five labels on social media data, the alignment of LLMs with continuous, ground-truth personality assessments derived from natural interactions is largely unexamined. To address this gap, we introduce a novel benchmark comprising semi-structured interview transcripts paired with validated continuous Big Five trait scores. Using this dataset, we systematically evaluate LLM performance across three paradigms: (1) zero-shot and chain-of-thought prompting with GPT-4.1 Mini, (2) LoRA-based fine-tuning applied to both RoBERTa and Meta-LLaMA architectures, and (3) regression using static embeddings from pretrained BERT and OpenAI's text-embedding-3-small. Our results reveal that all Pearson correlations between model predictions and ground-truth personality traits remain below 0.26, highlighting the limited alignment of current LLMs with validated psychological constructs. Chain-of-thought prompting offers minimal gains over zero-shot, suggesting that personality inference relies more on latent semantic representation than explicit reasoning. These findings underscore the challenges of aligning LLMs with complex human attributes and motivate future work on trait-specific prompting, context-aware modeling, and alignment-oriented fine-tuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
