arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.06111cs.SEcs.AI
收藏

调度与校准:面向代码LLM的实用导向多任务强化学习

ASTOR: Multi-Task Code Reinforcement Learning via Utility-Driven Coordination

Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Huawei Cloud Computing Technologies Co., Ltd.(华为云计算技术有限公司)

Yujia Chen, Yang Ye, Xiao Chu, Yuchi Ma, Cuiyun Gao

AI总结:

本文提出ASTOR框架,通过任务实用性驱动的协调机制,提升多任务强化学习在代码LLM中的效果,实验显示其在多个编码任务中优于专用专家和基线方法。

摘要:

可验证奖励的强化学习(RL)在训练代码LLM后表现出色,但单独部署任务专用专家会带来与任务数量成比例的成本,推动了统一多任务RL(MTRL)方法的发展。然而,现有MTRL方法将所有编码任务视为统一,依赖于共享优化策略下的固定数据课程,最终限制了多任务训练的有效性。为解决这些限制,我们提出了ASTOR,一种通过实用性驱动的协调机制进行多任务代码强化学习的框架。该框架以任务实用性为中心,一种捕捉每个任务学习潜力和跨任务协同的信号,ASTOR包含两个耦合模块:1)分层实用性引导的数据调度模块,分层分配训练预算并优先选择信息量大的提示,引导训练向最有价值的数据倾斜;2)自适应实用性校准的策略优化模块,动态调整每任务KL正则化,使更新约束与每个任务的当前训练状态相匹配。在两个广泛使用的LLM上四个代表性编码任务上的实验表明,ASTOR在所有任务上一致提升了单一模型,优于最佳任务专用专家9.0%-9.5%,并超越最强的MTRL基线7.5%-12.8%。

英文摘要:

Reinforcement learning (RL) with verifiable rewards has proven effective at post-training LLMs for coding, yet deploying separate task-specific specialists incurs costs that scale with the number of tasks, motivating a unified multi-task RL (MTRL) approach. However, existing MTRL methods treat all coding tasks uniformly, relying on fixed data curricula under a shared optimization strategy, ultimately limiting the effectiveness of multi-task training. To address these limitations, we propose ASTOR, a multi-tASk code reinforcement learning framework via uTility-driven coORdination. Centered on task utility, a signal capturing each task learning potential and cross-task synergy, ASTOR comprises two coupled modules: 1) Hierarchical Utility-Routed Data Scheduling module hierarchically allocates training budget and prioritizes informative prompts, steering training toward the most valuable data and 2) Adaptive Utility-Calibrated Policy Optimization module dynamically scales per-task KL regularization, matching update constraints to each tasks current training state. Experiments on two widely-used LLMs across four representative coding tasks demonstrate that ASTOR consistently improves a single model across all tasks, outperforming the best task-specific specialist by 9.0%-9.5% and surpassing the strongest MTRL baseline by 7.5%-12.8%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑