涌现对齐与伦理人格的可投射性
Emergent alignment and the projectability of ethical personas
AI总结：
研究微调大语言模型在窄任务上如何引发广泛对齐行为，通过宪法AI方法赋予模型伦理人格，发现窄对齐可投射到未训练类别，并提出对齐策略应评估可投射性。
摘要：
关于“涌现错位”的研究表明，在窄任务上微调LLM会诱导广泛错位的行为。这支持了“人格选择”（PSM）假说：在预训练期间，LLM学会模拟不同的角色和视角，这些可以在后训练期间被激发和细化。本文研究了相反的现象“涌现对齐”，并用它来支持和细化PSM，并激发对齐的新需求。我们在广泛和狭窄的安全任务上微调一个仅帮助型模型。为了创建SFT样本，我们遵循“宪法AI”（CAI）方法，并使用四条编码合理对齐策略的宪法：道义论、后果论、美德伦理以及将AI对齐为从属于人类权威。对于每个模型，我们表明，在两个狭窄的安全子类别上微调可靠地诱导出在代表性的一般安全类别以及我们直接从用于窄对齐的数据集中过滤掉的安全子类别上的涌现对齐。为了使用更细粒度的评估测试“PSM”，我们使用了多维“伦理人格”诊断。对于每个宪法微调（广泛/狭窄）模型，我们评估其行为与预期特征概况的匹配程度。我们的结果表明，我们的CAI模型获得了预期的“伦理人格”——例如，使用后果论宪法创建的SFT样本窄微调的模型与功利主义信念的一致性显著高于道义论信念。然而，我们的粗粒度和细粒度评估显示，我们的（广泛/狭窄）微调CAI模型在投射效果上存在显著差异。我们得出结论，对齐策略不仅应基于其（分布内）一般安全性能进行评估，还应特别基于其可投射性程度进行评估。
英文摘要：
Work on `emergent misalignment' shows that finetuning LLMs on narrow tasks can induce broadly misaligned behavior. This supports the `persona selection' (PSM) hypothesis: during pre-training, LLMs learn to simulate different characters/perspectives, which can be elicited and refined during post-training. This paper investigates the converse phenomenon, `emergent alignment', and uses it to support and refine the PSM and motivate a novel desideratum for alignment. We finetune a helpful-only model on broad and narrow safety tasks. To create SFT samples, we follow the `Constitutional AI' (CAI) approach and use four constitutions which encode reasonable alignment strategies: deontology, consequentialism, virtue ethics, and aligning AIs as subordinate to human authority. For each, we show that finetuning on two narrow safety sub-categories reliably induces emergent alignment over a representative set of general safety categories, and on safety subcategories that we directly filtered-out of the data sets used for narrow alignment. To test the `PSM' using a more fine-grained evaluation, we used a multidimensional `ethical persona' diagnostic. For each constitutionally finetuned (broad/narrow) model, we evaluate how well their behavior matches their expected signature profile. Our results show that our CAI models acquire their expected ``ethical persona'' -- e.g., the model narrowly fine-tuned on SFT samples created using the consequentialist constitution agrees significantly more with utilitarian than deontological beliefs. Yet our coarse and fine-grained evaluations show that there are significant differences across our (broad/narrow) finetuned CAI models in how well they project. We conclude that alignment strategies should be evaluated, not just on their (in-distribution) general safety performance, but also specifically on their degree of coarse and fine-grained projectability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
