SDA：无微调的开放语言模型分布对齐框架
SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuning
AI总结：
SDA 提出了一种无需微调的开源 LLMs 分布对齐框架，通过用户指令动态调整输出概率，提升模型与人类意图的一致性。
摘要：
随着大型语言模型（LLMs）的快速发展，其在现实应用中的部署已变得越来越广泛。LLMs 被期望在多样任务、用户偏好和实际场景中提供稳健的性能。然而，随着需求的增长，确保 LLMs 生成的响应与人类意图一致仍是一个基础挑战。特别是，在推理过程中有效且高效地对齐模型行为，而无需昂贵的重新训练或广泛的监督，既是关键要求，也是一个非平凡的技术挑战。为了解决这一挑战，我们提出了 SDA（Steering-Driven Distribution Alignment），一种无需训练且模型无关的对齐框架，专为开源 LLMs 设计。SDA 根据用户定义的对齐指令动态重新分配模型输出概率，增强模型行为与人类意图之间的对齐，而无需微调。该方法轻量、资源高效，并且兼容多种开源 LLMs。它可以在推理过程中独立运行，或与基于训练的对齐策略集成。此外，SDA 支持个性化偏好对齐，使模型响应行为具有灵活的控制。实验证明，SDA 在 8 个不同规模和来源的开源 LLMs 上，通过三个关键对齐维度（帮助性、无害性和诚实性）一致提高了对齐性能。具体而言，SDA 在测试模型上实现了平均帮助性提升 64.4%，诚实性提升 30%，无害性提升 11.5%，表明其在不同模型和应用场景中的有效性和通用性。
英文摘要：
With the rapid advancement of large language models (LLMs), their deployment in real-world applications has become increasingly widespread. LLMs are expected to deliver robust performance across diverse tasks, user preferences, and practical scenarios. However, as demands grow, ensuring that LLMs produce responses aligned with human intent remains a foundational challenge. In particular, aligning model behavior effectively and efficiently during inference, without costly retraining or extensive supervision, is both a critical requirement and a non-trivial technical endeavor. To address the challenge, we propose SDA (Steering-Driven Distribution Alignment), a training-free and model-agnostic alignment framework designed for open-source LLMs. SDA dynamically redistributes model output probabilities based on user-defined alignment instructions, enhancing alignment between model behavior and human intents without fine-tuning. The method is lightweight, resource-efficient, and compatible with a wide range of open-source LLMs. It can function independently during inference or be integrated with training-based alignment strategies. Moreover, SDA supports personalized preference alignment, enabling flexible control over the model response behavior. Empirical results demonstrate that SDA consistently improves alignment performance across 8 open-source LLMs with varying scales and diverse origins, evaluated on three key alignment dimensions, helpfulness, harmlessness, and honesty (3H). Specifically, SDA achieves average gains of 64.4% in helpfulness, 30% in honesty and 11.5% in harmlessness across the tested models, indicating its effectiveness and generalization across diverse models and application scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
