LLM个性诱导中的评估漂移：我们是否在移动目标？
Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?
AI总结：
本文研究了大型语言模型在诱导人类样性格时的稳定性问题，通过微调长格式论文来诱导性格，并发现尽管微调减少了问卷评分的方差，但完整五维性格的准确性仍接近随机，表明无指导的论文缺乏表达忠实性格所需的线索。
摘要：
大型语言模型能否可靠地表达人类般的性格，还是仅仅在模仿表面线索而缺乏稳定的底层特征？为探讨此问题，我们通过在长格式论文上进行微调来诱导LLM的性格，每篇论文都关联一个目标大五人格特征轮廓。随后，我们使用IPIP-NEO问卷评估诱导性格的稳定性和准确性。具体而言，我们提出两个问题：（i）训练后（SFT、DPO、ORPO）是否在提示重新表述下稳定问卷评分？（ii）能否从无指导的论文中诱导目标大五人格特征？我们的结果表明，微调在五个模型上一致减少了问卷响应的方差，直接缓解了预训练模型中报告的评估脆弱性。然而，这种新发现的稳定性揭示了更根本的限制：即使单个特征分数有所提高，完整五维特征的准确性仍接近随机。这表明无指导的论文缺乏表达忠实性格所需的线索。因此，我们主张使用场景相关的数据集或交互式引导来积累与测试一致的证据。
英文摘要：
Can large language models reliably express a human-like personality, or are they merely mimicking surface cues without a stable underlying profile? To investigate this, we induce personality in LLMs by fine-tuning them on the long-form essays, where each essay is associated with a target Big Five personality profile. We then evaluate the stability and fidelity of the induced personality using the IPIP-NEO questionnaire. Specifically, we ask: (i) does post-training (SFT, DPO, ORPO) stabilize questionnaire scores under prompt rephrasings, and (ii) can it induce target Big Five profiles from unguided essays? Our results demonstrate that fine-tuning consistently reduces variance in questionnaire responses across five models, directly mitigating the evaluation fragility reported in pre-trained models. However, this newfound stability reveals a more fundamental limitation: accuracy on the full five-dimensional profile remains near chance, even when single-trait scores improve. This indicates that unguided essays lack the cues needed for faithful personality expression. We therefore argue for scenario-grounded datasets or interactive elicitation that accumulates test-aligned evidence over time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
