Persona Vectors：监测与控制语言模型中的性格特征
Persona Vectors: Monitoring and Controlling Character Traits in Language Models
AI总结：
该研究提出人设向量方法，可自动提取对应不同性格特征的激活空间方向，用于监测模型性格波动、预测控制训练中的性格转变，还能标记不良训练数据，实现对语言模型性格特征的有效管控。
摘要：
大语言模型通过模拟的“助手”人设与用户交互。尽管助手通常被训练为有帮助、无害且诚实的，但它有时会偏离这些理想状态。在本文中，我们识别出模型激活空间中支撑若干特征（如邪恶、谄媚、幻觉倾向）的方向——persona vectors（人设向量）。我们证实这些向量可用于在部署阶段监测助手性格的波动。随后我们将人设向量应用于预测和控制训练过程中出现的性格转变。我们发现微调后预期和非预期的性格变化都与沿相关人设向量的偏移高度相关。这些偏移可通过事后干预得到缓解，或通过一种新的预防式引导方法从一开始就被避免。此外，人设向量可用于标记会产生不良性格变化的训练数据，涵盖数据集层面和单个样本层面。我们提取人设向量的方法是自动化的，仅需自然语言描述即可应用于任何感兴趣的性格特征。
英文摘要：
Large language models interact with users through a simulated 'Assistant' persona. While the Assistant is typically trained to be helpful, harmless, and honest, it sometimes deviates from these ideals. In this paper, we identify directions in the model's activation space-persona vectors-underlying several traits, such as evil, sycophancy, and propensity to hallucinate. We confirm that these vectors can be used to monitor fluctuations in the Assistant's personality at deployment time. We then apply persona vectors to predict and control personality shifts that occur during training. We find that both intended and unintended personality changes after finetuning are strongly correlated with shifts along the relevant persona vectors. These shifts can be mitigated through post-hoc intervention, or avoided in the first place with a new preventative steering method. Moreover, persona vectors can be used to flag training data that will produce undesirable personality changes, both at the dataset level and the individual sample level. Our method for extracting persona vectors is automated and can be applied to any personality trait of interest, given only a natural-language description.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
