开放角色训练：通过Constitutional AI塑造AI助手的人格
Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI
AI总结：
本研究提出了首个基于Constitutional AI和合成内省数据的角色训练开放实现，通过微调模型有效且可控地塑造AI助手人格，在提升对抗鲁棒性与生成真实性的同时保持通用能力不变。
摘要：
现代聊天机器人大型语言模型生成的“AI助手”人格特征会影响其表层行为以及显现的价值观、信念和伦理。这些都会影响交互质量、感知智能度，以及与开发者和用户意图的对齐程度。这种人格的塑造被称为角色训练，是工业界后训练的关键组成部分，但在学术文献中实际上尚未被研究。我们引入了角色训练的首个开放实现，利用Constitutional AI和一条使用合成内省数据的新数据管道，以比约束系统提示或激活引导等替代方案更有效、更可控的方式塑造助手人格。具体而言，我们使用11个示例人格（如幽默、深切关怀甚至恶意）微调了三个流行的开放权重模型。为了追踪我们方法的效果，我们引入了一种分析显示偏好的方法，揭示了角色上清晰且整体的变化。我们发现，与上述两种替代方案相比，这些变化在面对对抗性提示时更具鲁棒性，同时也能产生更连贯、更真实的生成内容。最后，我们证明这种微调对常见基准测试衡量的通用能力几乎没有影响。我们描述并开源了完整的后训练方法，其实现可在 https://github.com/maiush/OpenCharacterTraining 找到。
英文摘要：
The character of the "AI assistant" persona generated by modern chatbot large language models influences both surface-level behavior and apparent values, beliefs, and ethics. These all affect interaction quality, perceived intelligence, and alignment with both developer and user intentions. The shaping of this persona, known as character training, is a critical component of industry post-training, yet remains effectively unstudied in the academic literature. We introduce the first open implementation of character training, leveraging Constitutional AI and a new data pipeline using synthetic introspective data to shape the assistant persona in a more effective and controlled manner than alternatives such as constraining system prompts or activation steering. Specifically, we fine-tune three popular open-weights models using 11 example personas, such as humorous, deeply caring, or even malevolent. To track the effects of our approach, we introduce a method which analyzes revealed preferences, uncovering clear and holistic changes in character. We find these changes are more robust to adversarial prompting than the above two alternatives, while also leading to more coherent and realistic generations. Finally, we demonstrate this fine-tuning has little to no effect on general capabilities as measured by common benchmarks. We describe and open-source our full post-training method, the implementation of which can be found at https://github.com/maiush/OpenCharacterTraining.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
