YaPO: 用于领域适应的可学习稀疏激活引导向量
YaPO: Learnable Sparse Activation Steering Vectors for Domain Adaptation
AI总结：
YaPO通过学习稀疏激活引导向量实现LLM的高效、稳定和细粒度对齐，适用于领域适应和文化对齐等任务。
摘要：
通过激活干预来指导大型语言模型（LLMs）已成为一种轻量级替代方法，用于对齐和个性化。最近的双向偏好优化（BiPO）研究表明，可以通过直接偏好优化（DPO）的方式从偏好数据中直接学习密集引导向量，从而控制诚实性、幻觉和安全性行为。然而，密集引导向量往往由于神经元多语义性而纠缠多个潜在因素，限制了它们在细粒度设置中的有效性和稳定性，例如文化对齐，其中必须区分密切相关的价值观和行为（例如中东文化之间的差异）。在本文中，我们提出了Yet another Policy Optimization（YaPO），一种无参考方法，它在稀疏编码器（SAE）的潜在空间中学习稀疏引导向量。通过优化稀疏代码，YaPO生成解耦、可解释且高效的引导方向。实证研究表明，YaPO比密集引导基线收敛更快，性能更强，且训练稳定性更高。除了文化对齐外，YaPO还推广到一系列与对齐相关的行为，包括幻觉、财富追求、越狱和权力追求。重要的是，YaPO保留了通用知识，在MMLU上没有可测量的降级。总体而言，我们的结果表明，YaPO为LLMs的高效、稳定和细粒度对齐提供了一种通用方法，具有广泛的应用于可控性和领域适应。相关代码和数据已公开\footnote{https://github.com/MBZUAI-Paris/YaPO}。
英文摘要：
Steering Large Language Models (LLMs) through activation interventions has emerged as a lightweight alternative to fine-tuning for alignment and personalization. Recent work on Bi-directional Preference Optimization (BiPO) shows that dense steering vectors can be learned directly from preference data in a Direct Preference Optimization (DPO) fashion, enabling control over truthfulness, hallucinations, and safety behaviors. However, dense steering vectors often entangle multiple latent factors due to neuron multi-semanticity, limiting their effectiveness and stability in fine-grained settings such as cultural alignment, where closely related values and behaviors (e.g., among Middle Eastern cultures) must be distinguished. In this paper, we propose Yet another Policy Optimization (YaPO), a \textit{reference-free} method that learns \textit{sparse steering vectors} in the latent space of a Sparse Autoencoder (SAE). By optimizing sparse codes, YaPO produces disentangled, interpretable, and efficient steering directions. Empirically, we show that YaPO converges faster, achieves stronger performance, and exhibits improved training stability compared to dense steering baselines. Beyond cultural alignment, YaPO generalizes to a range of alignment-related behaviors, including hallucination, wealth-seeking, jailbreak, and power-seeking. Importantly, YaPO preserves general knowledge, with no measurable degradation on MMLU. Overall, our results show that YaPO provides a general recipe for efficient, stable, and fine-grained alignment of LLMs, with broad applications to controllability and domain adaptation. The associated code and data are publicly available\footnote{https://github.com/MBZUAI-Paris/YaPO}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
