通过方向邻域共识实现鲁棒偏好对齐
Robust Preference Alignment via Directional Neighborhood Consensus
AI总结：
通过方向邻域共识实现鲁棒偏好对齐，提升模型在多样化偏好下的稳定性与可靠性。
摘要：
将大型语言模型与人类偏好对齐对于创建可靠且可控的AI系统至关重要。人类偏好可以可视化为一个高维向量，其中不同方向代表不同属性之间的权衡（例如，有用性与冗长性）。然而，由于训练数据通常反映主导的平均偏好，LLM在处理常见请求时表现良好，但在特定、个体需求上则表现不佳。这种不匹配导致了偏好覆盖的缺口。现有方法通常通过昂贵的重新训练来解决这个问题，这可能无法推广到广泛的多样化偏好。这种脆弱性意味着当用户的请求反映了一种偏离训练数据主流的细微偏好时，模型性能可能会不可预测地下降。为了解决这一挑战，我们引入了鲁棒偏好选择（RPS），这是一种后处理、无训练的方法，通过利用方向邻域共识。与迫使模型从单一高度特定的偏好生成响应不同，RPS从相关偏好的局部邻域中采样多个响应，生成一个更优的候选池。然后选择最符合用户原始意图的响应。我们提供了一个理论框架，证明我们的邻域生成策略在理论上优于一个也采样多个候选的强基线。在三种不同的对齐范式（DPA、DPO和SFT）上进行的全面实验表明，RPS在对抗这种基线时表现出一致的鲁棒性，能够在挑战性偏好中从欠代表区域的空间中实现高达69%的胜率，而无需任何模型重新训练。我们的工作为增强偏好对齐模型的可靠性提供了一种实用且理论严谨的解决方案。
英文摘要：
Aligning large language models with human preferences is critical for creating reliable and controllable AI systems. A human preference can be visualized as a high-dimensional vector where different directions represent trade-offs between desired attributes (e.g., helpfulness vs. verbosity). Yet, because the training data often reflects dominant, average preferences, LLMs tend to perform well on common requests but fall short in specific, individual needs. This mismatch creates a preference coverage gap. Existing methods often address this through costly retraining, which may not be generalized to the full spectrum of diverse preferences. This brittleness means that when a user's request reflects a nuanced preference deviating from the training data's central tendency, model performance can degrade unpredictably. To address this challenge, we introduce Robust Preference Selection (RPS), a post-hoc, training-free method by leveraging directional neighborhood consensus. Instead of forcing a model to generate a response from a single, highly specific preference, RPS samples multiple responses from a local neighborhood of related preferences to create a superior candidate pool. It then selects the response that best aligns with the user's original intent. We provide a theoretical framework showing our neighborhood generation strategy is provably superior to a strong baseline that also samples multiple candidates. Comprehensive experiments across three distinct alignment paradigms (DPA, DPO, and SFT) demonstrate that RPS consistently improves robustness against this baseline, achieving win rates of up to 69% on challenging preferences from under-represented regions of the space without any model retraining. Our work presents a practical, theoretically-grounded solution for enhancing the reliability of preference-aligned models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
