APPA：自适应偏好多元对齐用于大语言模型公平联邦强化学习从人类反馈
APPA: Adaptive Preference Pluralistic Alignment for Fair Federated RLHF of LLMs
AI总结：
本文提出APPA框架，通过动态调整群体奖励权重，提升联邦强化学习中多群体公平对齐效果，实验显示其在保持整体对齐性的同时，显著提升最差群体对齐性能。
摘要：
对齐大型语言模型（LLMs）与多样化的用户偏好需要多元对齐，即单个模型必须同时尊重多个不同群体的价值。在联邦强化学习从人类反馈（FedRLHF）中，这些群体通过共享策略对齐，而无需集中偏好数据，这使得公平奖励聚合至关重要。现有聚合方法存在明显权衡：基于平均的聚合系统性地低估了表现最差的群体，而基于最小值的聚合则优先考虑最差群体性能，以牺牲整体对齐性为代价。我们提出APPA，一种自适应偏好多元对齐框架，根据历史对齐奖励动态调整群体层面的奖励权重。我们的方法优先考虑未对齐的群体，而不影响表现良好的群体，同时无需访问原始偏好数据。将APPA集成到基于近端策略优化（PPO）的FedRLHF流程中，并在GLOBALQA和OQA上跨三个模型家族（Gemma 2 2B、Llama 3.2 3B、Qwen3 0.6B）进行评估，APPA实现了强大的公平对齐权衡，比基于平均的聚合方法提高了最差群体对齐性能高达28%，同时在大多数配置中保持比基于最小值的聚合方法更高的整体对齐性。
英文摘要：
Aligning large language models (LLMs) with diverse human preferences requires pluralistic alignment, where a single model must respect the values of multiple distinct groups simultaneously. In federated reinforcement learning from human feedback (FedRLHF), these groups align a shared policy without centralizing preference data, which makes fair reward aggregation essential. Existing aggregation methods exhibit clear trade offs: average based aggregation systematically under aligns worst performing groups, while min aggregation prioritizes worst group performance at the cost of overall alignment. We propose APPA, an Adaptive Preference Pluralistic Alignment framework that dynamically reweights group level rewards based on historical alignment rewards. Our approach prioritizes under aligned groups without degrading well aligned ones, while requiring no access to raw preference data. Integrated into a proximal policy optimization (PPO) based FedRLHF pipeline and evaluated on GLOBALQA and OQA across three model families (Gemma 2 2B, Llama 3.2 3B, Qwen3 0.6B), APPA achieves strong fairness alignment trade offs, improving worst group alignment by up to 28% over average aggregation while maintaining higher overall alignment than min aggregation across most configurations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
