我们思考，因此我们对LLMs进行对齐，使其在出错前变得有益、无害和诚实
We Think, Therefore We Align LLMs to Helpful, Harmless and Honest Before They Go Wrong
AI总结：
本文提出AMBS框架，通过在1-to-N Transformer设置中参数化目标特定转换，解决LLM对齐中多个目标之间的干扰问题，提升HHH目标的联合满足能力。
摘要：
大语言模型（LLM）的对齐能力是指在生成过程中满足预期目标的能力，这对于可信部署至关重要。在实践中，对齐通常通过多个目标如Helpfulness、Harmlessness和Honesty（HHH）来操作化。先前的工作通过在标准Transformer解码器中使用引导向量来研究对齐，但将目标孤立处理，优化单个目标会覆盖其他目标，导致干扰。近期的工作尝试通过将引导扩展到1-to-N Transformer设置来解决这一限制，通过将表示复制到目标特定路径中，但应用转换独立，导致目标之间响应不一致。同样，安全RLHF和MoE设计方法研究目标之间的权衡，但不约束目标特定转换在共享表示中的限制。因此，即使是最先进的LLM在复杂情况下也难以同时满足HHH目标。为此，我们提出自适应多分支引导（AMBS），一种在1-to-N Transformer设置中的两阶段框架，参数化目标特定转换相对于共享表示。在第一阶段，计算一次共享隐藏表示。在第二阶段，该表示被复制到N条路径中，并相对于共享参考更新，捕捉目标特定偏差的同时限制分歧。这在单次前向传递中产生N个目标特定响应，可以在解码时结合以获得跨目标的单一响应。在多个主干网络上，AMBS在HHH目标上提升了性能，具有在WR、TI和SS（例如，LLaMA-2-7B上平均56.5%）上的一致提升，同时保持效率（例如，189 Tok/s，9 GPU-hrs）
英文摘要：
Alignment of Large Language Models (LLMs) is the ability to satisfy desired objectives during generation, which is critical for trustworthy deployment. In practice, alignment is often operationalized through multiple objectives such as Helpfulness, Harmlessness, and Honesty (HHH). Prior works study alignment via steering vectors in standard Transformer decoders but treat objectives in isolation, where optimizing a single objective can overwrite others, leading to interference. Recent works attempt to address this limitation by extending steering to a 1-to-N Transformer setting by replicating representations into objective-specific pathways, but apply transformations independently, resulting in inconsistent responses across objectives. Similarly, approaches such as safe RLHF and MoE-based designs study trade-offs across objectives but do not constrain objective-specific transformations within a shared representation during inference. As a result, even aligned State-of-the-Art (SOTA) LLMs can struggle to jointly satisfy HHH objectives in complex settings. To address this, we propose Adaptive Multi-Branch Steering (AMBS), a two-stage framework in a 1-to-N Transformer setting that parameterizes objective-specific transformations relative to a shared representation. In Stage I, a shared hidden representation is computed once. In Stage II, this representation is replicated into N pathways and updated relative to a shared reference, capturing objective-specific deviations while restricting divergence. This produces N objective-specific responses within a single forward pass, which can be combined at decoding to obtain a single response across objectives. Across multiple backbones, AMBS improves performance across HHH, with consistent gains in WR, TI, and SS (e.g., Avg 56.5% on LLaMA-2-7B) while maintaining efficiency (e.g., 189 Tok/s, 9 GPU-hrs).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
