arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.01735cs.CLcs.LG
收藏

LASeR:基于多臂老虎机的奖励模型自适应选择学习方法

LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Bandits

UNC Chapel Hill(杜克大学夏洛特分校);The University of Texas at Austin(德克萨斯大学奥斯汀分校)

Duy Nguyen, Archiki Prasad, Elias Stengel-Eskin, Mohit Bansal

AI总结:

针对单固定RM训练LLM泛化不足、多RM并行训练成本高且信号冲突的问题,提出将RM选择建模为多臂老虎机的LASeR方法,在多类任务上均提升了性能与效率。

摘要:

奖励模型(Reward Models, RMs)对大语言模型(large language models, LLMs)的对齐至关重要,但针对某一任务(如写作)专门优化的RM在新任务(如数学)上的泛化能力通常无法预先获知,这往往导致仅用单个固定RM训练LLM的效果并非最优。然而,同时使用多个RM优化LLM会产生极高的计算成本,且不同RM的信号可能相互冲突,反而会降低性能。为解决这些挑战,我们提出LASeR(Learning to Adaptively Select Rewards,自适应奖励选择学习方法),该方法将奖励模型选择问题建模为多臂老虎机问题,通过为每个样本选择最合适的RM,高效迭代地训练LLM。在常识推理和数学推理任务上,我们验证了LASeR能提升LLM的迭代训练效果:在三个数据集上,Llama-3-8B的平均绝对准确率比RM分数集成方法高2.67%,同时效率更优(例如速度提升2倍)。此外,在WildChat(开放式指令遵循任务)上,LASeR的AlpacaEval胜率比RM分数集成基线高72.69%。扩展到长上下文生成场景时,在单文档问答任务上LASeR的平均F1值比采用best-of-n采样的RM分数集成基线高2.96,在少样本学习任务上高2.97。

英文摘要:

Reward Models (RMs) are crucial to aligning large language models (LLMs), but the degree to which an RM specialized to one task (e.g. writing) generalizes to new tasks (e.g. math) is often not known a priori, often making using only one fixed RM to train LLMs suboptimal. However, optimizing LLMs with multiple RMs simultaneously can incur a prohibitively high computational cost and lead to conflicting signals from different RMs that may degrade performance. To address these challenges, we introduce LASeR (Learning to Adaptively Select Rewards), which frames reward model selection as a multi-armed bandit problem, efficiently and iteratively training LLMs using multiple RMs by selecting the most well-suited RM for each instance. On commonsense and math reasoning tasks, we show that LASeR boosts iterative LLM training, improving the absolute average accuracy of Llama-3-8B over three datasets by 2.67% over an ensemble of RM scores while also showing superior efficiency (e.g., a 2x speedup). Moreover, on WildChat (open-ended instruction-following tasks), LASeR leads to a 72.69% AlpacaEval win rate over the RM score ensemble baseline. Extending to long-context generation, LASeR improves by 2.96 F1 points (avg.) on single-document QA tasks and 2.97 F1 points on few-shot learning over the RM score ensemble baseline with best-of-n sampling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑