arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.05965cs.AI
收藏

超越妥协:用于高效多偏好LLM对齐的帕累托宽容共识

Beyond Compromise: Pareto-Lenient Consensus for Efficient Multi-Preference LLM Alignment

Zhejiang Lab(之江实验室);Macau University of Science and Technology(澳门科技大学)

Renxuan Tan, Rongpeng Li, Zhifeng Zhao, Honggang Zhang

AI总结:

本文提出帕累托宽容共识框架,通过动态谈判过程解决多目标偏好对齐中的局部收敛问题,提升模型对全局帕累托最优前沿的探索能力。

摘要:

超越单一偏好范式,对齐LLM与多样化的人类价值观对稳健部署至关重要。当前多目标偏好对齐(MPA)方法主要依赖静态线性标量化或刚性梯度投影来处理这些权衡。然而,通过强制严格冲突回避或同时下降,这些范式常过早收敛到局部 stationary 点。虽然数学上稳定,这些点代表保守妥协,模型牺牲潜在全局帕累托改进以避免短暂局部权衡。为打破这一僵局,我们提出帕累托宽容共识(PLC),一种博弈论框架,将对齐重新想象为动态谈判过程。不同于刚性方法,PLC 引入共识驱动的宽容梯度校正,动态容忍局部退化,只要存在足够的主导联盟盈余,从而赋能优化轨迹逃离局部次优均衡并探索远端帕累托最优前沿。理论分析验证PLC可促进僵局突破并渐近收敛到帕累托共识均衡。此外,广泛实验表明PLC在固定偏好对齐和全局帕累托前沿质量上均优于基线。本文强调谈判驱动对齐作为MPA的潜在前景。我们的代码可在https://anonymous.4open.science/r/aaa-6BB8获取。

英文摘要:

Transcending the single-preference paradigm, aligning LLMs with diverse human values is pivotal for robust deployment. Contemporary Multi-Objective Preference Alignment (MPA) approaches predominantly rely on static linear scalarization or rigid gradient projection to navigate these trade-offs. However, by enforcing strict conflict avoidance or simultaneous descent, these paradigms often prematurely converge to local stationary points. While mathematically stable, these points represent a conservative compromise where the model sacrifices potential global Pareto improvements to avoid transient local trade-offs. To break this deadlock, we propose Pareto-Lenient Consensus (PLC), a game-theoretic framework that reimagines alignment as a dynamic negotiation process. Unlike rigid approaches, PLC introduces consensus-driven lenient gradient rectification, which dynamically tolerates local degradation provided there is a sufficient dominant coalition surplus, thereby empowering the optimization trajectory to escape local suboptimal equilibrium and explore the distal Pareto-optimal frontier. Theoretical analysis validates PLC can facilitate stalemate escape and asymptotically converge to a Pareto consensus equilibrium. Moreover, extensive experiments show that PLC surpasses baselines in both fixed-preference alignment and global Pareto frontier quality. This work highlights the potential of negotiation-driven alignment as a promising avenue for MPA. Our codes are available at https://anonymous.4open.science/r/aaa-6BB8.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑