arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2509.04794cs.CL

人格作为大语言模型评估探针：方法权衡与下游效应

Personality as a Probe for LLM Evaluation: Method Trade-offs and Downstream Effects

University College London（伦敦大学学院）

Gunmay Handa, Zekun Wu, Adriano Koshiyama, Philip Treleaven

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

该研究以大五人格为探针系统对比ICL、PEFT、MS三种LLM人格控制方法，构建对比数据集与统一评估框架，提出特质纯化技术和三级稳定性框架，明确各方法的效果与能力权衡，为模型部署和可解释性研究提供新思路。

摘要：

大语言模型（LLM）中的人格操控正越来越多地应用于客户服务和智能体场景，但其作用机制与权衡关系仍不明确。我们针对基于大五人格特质的人格控制开展了系统性研究，对比了上下文学习（ICL）、参数高效微调（PEFT）和机制引导（MS）三种方法。我们的贡献分为四个方面：第一，构建了一个包含平衡的高/低特质响应的对比数据集，可支持有效的引导向量计算和公平的跨方法评估；第二，提出了基于运行内Δ分析的统一评估框架，在MMLU、GAIA和BBQ基准上分离衡量推理能力、智能体性能和人口统计偏见；第三，开发了特质纯化技术，用于分离开放性与尽责性，解决特质编码中的表征重叠问题；第四，提出了三级稳定性框架，量化方法级、特质级和组合级的鲁棒性，为部署约束下的实践提供指导。在Gemma-2-2B-IT和LLaMA-3-8B-Instruct上的实验展现出明确的权衡关系：ICL实现了较强的对齐效果且能力损失最小，PEFT的对齐效果最高但会降低任务性能，MS则提供了轻量级的运行时控制且效果具有竞争力。特质级分析显示，开放性是唯一极具挑战性的特质，宜人性对ICL的抗性最强，人格编码主要集中在中间层。综上，这些结果确立了人格操控可作为行为表征的多层次探针，连接了表层条件作用、参数编码和激活层引导，并表明机制引导是部署和可解释性场景中微调的轻量级替代方案。

英文摘要：

Personality manipulation in large language models (LLMs) is increasingly applied in customer service and agentic scenarios, yet its mechanisms and trade-offs remain unclear. We present a systematic study of personality control using the Big Five traits, comparing in-context learning (ICL), parameter-efficient fine-tuning (PEFT), and mechanistic steering (MS). Our contributions are fourfold. First, we construct a contrastive dataset with balanced high/low trait responses, enabling effective steering vector computation and fair cross-method evaluation. Second, we introduce a unified evaluation framework based on within-run $Δ$ analysis that disentangles, reasoning capability, agent performance, and demographic bias across MMLU, GAIA, and BBQ benchmarks. Third, we develop trait purification techniques to separate openness from conscientiousness, addressing representational overlap in trait encoding. Fourth, we propose a three-level stability framework that quantifies method-, trait-, and combination-level robustness, offering practical guidance under deployment constraints. Experiments on Gemma-2-2B-IT and LLaMA-3-8B-Instruct reveal clear trade-offs: ICL achieves strong alignment with minimal capability loss, PEFT delivers the highest alignment at the cost of degraded task performance, and MS provides lightweight runtime control with competitive effectiveness. Trait-level analysis shows openness as uniquely challenging, agreeableness as most resistant to ICL, and personality encoding consolidating around intermediate layers. Taken together, these results establish personality manipulation as a multi-level probe into behavioral representation, linking surface conditioning, parameter encoding, and activation-level steering, and positioning mechanistic steering as a lightweight alternative to fine-tuning for both deployment and interpretability.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑