DARC:通过风险约束解码实现的分歧意识对齐
DARC: Disagreement-Aware Alignment via Risk-Constrained Decoding
AI总结:
DARC通过风险约束解码方法,在不重新训练的情况下,通过最大化KL-鲁棒满意度目标来缓解分歧和尾部风险,保持高质量输出。
摘要:
基于偏好对齐的方法(如RLHF、DPO)通常优化单一标量目标,隐式地平均异质人类偏好。在实践中,系统标注者和用户组的分歧使均值奖励最大化变得脆弱且易受代理过优化影响。我们提出了**通过风险约束解码实现的分歧意识对齐(DARC)**,一种无需重新训练的推理时间方法,将响应选择框架为分布鲁棒、风险敏感的决策制定。给定多个偏好样本或可扩展的分歧代理,DARC通过最大化KL-鲁棒(熵)满意度目标对候选者进行重新排序,并提供简单的部署控制,使相应的熵风险溢价相对于均值进行限制或惩罚,从而在不重新训练的情况下实现显式风险预算。我们提供了将此解码规则与原则性悲观主义和基于KL的分布鲁棒优化联系起来的理论分析。在对齐基准测试中,DARC在减少分歧和尾部风险的同时,保持在噪声、异质反馈下的竞争力平均质量。
英文摘要:
Preference-based alignment methods (e.g., RLHF, DPO) typically optimize a single scalar objective, implicitly averaging over heterogeneous human preferences. In practice, systematic annotator and user-group disagreement makes mean-reward maximization brittle and susceptible to proxy over-optimization. We propose **Disagreement-Aware Alignment via Risk-Constrained Decoding (DARC)**, a retraining-free inference-time method that frames response selection as distributionally robust, risk-sensitive decision making. Given multiple preference samples or scalable disagreement proxies, DARC reranks candidates by maximizing a *KL-robust (entropic)* satisfaction objective, and provides simple deployment controls that cap or penalize the corresponding entropic risk premium relative to the mean, enabling explicit risk budgets without retraining. We provide theoretical characterization linking this decoding rule to principled pessimism and KL-based distributionally robust optimization. Experiments on alignment benchmarks show that DARC reduces disagreement and tail risk while maintaining competitive average quality under noisy, heterogeneous feedback.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。