语言模型中引导方法的对齐性与有用性之间的权衡
Tradeoffs Between Alignment and Helpfulness in Language Models with Steering Methods
AI总结：
本文研究表征工程在对齐大语言模型时，对齐性提升与有用性下降的权衡，提出理论界限并实证验证，发现有用性二次方受损而对齐性线性提升。
摘要：
语言模型对齐已成为AI安全的重要组成部分，通过增强期望行为并抑制非期望行为，实现人类与语言模型之间的安全交互。对齐通常通过调整模型或插入预设的对齐提示来完成。近期，表征工程——一种通过在后训练阶段改变模型表征来改变模型行为的方法——被证明能有效对齐大语言模型（Zou等，2023a）。表征工程在对齐导向任务（如抵抗对抗性攻击和减少社会偏见）上取得了收益，但也导致模型执行基本任务的能力下降。本文研究模型对齐性提升与有用性下降之间的权衡。我们提出了一个理论框架，为这两个量提供界限，并实证验证其相关性。首先，我们发现在我们框架的条件下，表征工程可以保证对齐性，但同时有用性在此过程中受到损害。其次，我们证明有用性随表征工程向量的范数呈二次方下降，而对齐性随其线性增加，这表明存在一个使用表征工程高效的区间。我们实证验证了我们的发现，并描绘了表征工程用于对齐的有效性边界。
英文摘要：
Language model alignment has become an important component of AI safety, allowing safe interactions between humans and language models, by enhancing desired behaviors and inhibiting undesired ones. It is often done by tuning the model or inserting preset aligning prompts. Recently, representation engineering, a method which alters the model's behavior via changing its representations post-training, was shown to be effective in aligning LLMs (Zou et al., 2023a). Representation engineering yields gains in alignment oriented tasks such as resistance to adversarial attacks and reduction of social biases, but was also shown to cause a decrease in the ability of the model to perform basic tasks. In this paper we study the tradeoff between the increase in alignment and decrease in helpfulness of the model. We propose a theoretical framework which provides bounds for these two quantities, and demonstrate their relevance empirically. First, we find that under the conditions of our framework, alignment can be guaranteed with representation engineering, and at the same time that helpfulness is harmed in the process. Second, we show that helpfulness is harmed quadratically with the norm of the representation engineering vector, while the alignment increases linearly with it, indicating a regime in which it is efficient to use representation engineering. We validate our findings empirically, and chart the boundaries to the usefulness of representation engineering for alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
