对齐中的奖励模型路由
Reward Model Routing in Alignment
AI总结：
针对大语言模型对齐中单一奖励模型的局限及现有路由方法的冷启动等问题，提出结合离线优势学习与在线贝叶斯选择的BayesianRouter框架，在多个基准上性能优于现有方案。
摘要：
基于人类或AI反馈的强化学习（RLHF / RLAIF）已成为大语言模型（LLM）对齐的标准范式。然而，大多数流程依赖单个奖励模型（RM），限制了对齐质量且存在过拟合风险。近期研究探索了RM路由——从候选池中动态选择RM以利用互补优势，同时保持O(1)的RM调用次数——但现有方法存在冷启动和探索不足的问题。我们提出BayesianRouter，这是一种结合离线RM优势学习与在线贝叶斯选择的混合路由框架。在离线阶段，基于偏好数据训练多任务路由器以估计每个RM的可靠性。在在线阶段，贝叶斯汤普森采样路由器执行逐查询RM选择，以离线嵌入作为高斯先验初始化RM专属权重向量，并利用在线奖励自适应更新其后验分布，以适应不断演化的策略分布。在指令跟随（AlpacaEval-2、Arena-Hard、MT-Bench）和推理（GSM8K、MMLU）基准上的大量实验表明，BayesianRouter的性能始终优于单个RM、RM集成以及现有路由方法。
英文摘要：
Reinforcement learning from human or AI feedback (RLHF / RLAIF) has become the standard paradigm for aligning large language models (LLMs). However, most pipelines rely on a single reward model (RM), limiting alignment quality and risking overfitting. Recent work explores RM routing--dynamically selecting an RM from a candidate pool to exploit complementary strengths while maintaining $O(1)$ RM calls--but existing methods suffer from cold-start and insufficient exploration. We propose BayesianRouter, a hybrid routing framework that combines offline RM strengths learning with online Bayesian selection. In the offline stage, a multi-task router is trained on preference data to estimate per-RM reliability. In the online stage, a Bayesian Thompson sampling router performs per-query RM selection, initializing RM-specific weight vectors with offline embeddings as Gaussian priors and adaptively updating their posteriors with online rewards to adapt to the evolving policy distribution. Extensive experiments on instruction-following (AlpacaEval-2, Arena-Hard, MT-Bench) and reasoning (GSM8K, MMLU) benchmarks show that BayesianRouter consistently outperforms individual RMs, RM ensembling, and existing routing methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
