arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.15449cs.CLcs.LGcs.SE
收藏

TAROT: 为基于大语言模型的代码生成设计的测试驱动和能力适应课程强化微调

TAROT: Test-driven and Capability-adaptive Curriculum Reinforcement Fine-tuning for Code Generation with Large Language Models

Electronics and Telecommunications Research Institute(电信研究所);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学);Hugging Face;Ant Group(蚂蚁集团)

Chansung Park, Juyong Jiang, Fan Wang, Sayak Paul, Jiasi Shen, Jing Tang, Jianguo Li

AI总结:

TAROT通过测试驱动和能力适应的课程强化微调方法,提升大语言模型生成代码的功能正确性和稳健性。

摘要:

大型语言模型(LLMs)正在改变编程范式,即称为vibe编程,但生成算法复杂且稳健的代码仍然是一个关键挑战。激励LLMs的深层推理能力对于克服这一障碍至关重要。强化微调(RFT)已出现作为解决这一需求的有前途的策略。然而,大多数现有方法忽略了测试用例中固有的异质难度和粒度,导致奖励信号分布不平衡,从而在训练过程中产生偏差的梯度更新。为此,我们提出了测试驱动和能力适应课程强化微调(TAROT)。TAROT系统地为每个问题构建四层测试套件(基础、中间、复杂、边缘),为课程设计和评估提供受控难度景观。至关重要的是,TAROT将课程进度与原始奖励分数解耦,使能力条件下的评估和从课程策略组合中进行有原则的选择,而不是偶然的测试用例难度组成。这种设计促进了稳定的优化和更高效的技能获取。广泛的实验结果表明,RFT在代码生成中的最佳课程与模型的固有能力密切相关,能力较弱的模型在从简单到复杂的课程中获得更大的收益,而更擅长的模型则在先难后易的课程中表现优异。TAROT提供了一种可复现的方法,能够根据模型的能力自适应地调整课程设计,从而一致地提高生成代码的功能正确性和稳健性。所有代码和数据都已发布以促进可复现性和推动社区研究,网址为https://github.com/deep-diver/TAROT。

英文摘要:

Large Language Models (LLMs) are changing the coding paradigm, known as vibe coding, yet synthesizing algorithmically sophisticated and robust code still remains a critical challenge. Incentivizing the deep reasoning capabilities of LLMs is essential to overcoming this hurdle. Reinforcement Fine-Tuning (RFT) has emerged as a promising strategy to address this need. However, most existing approaches overlook the heterogeneous difficulty and granularity inherent in test cases, leading to an imbalanced distribution of reward signals and consequently biased gradient updates during training. To address this, we propose Test-driven and cApability-adaptive cuRriculum reinfOrcement fine-Tuning (TAROT). TAROT systematically constructs, for each problem, a four-tier test suite (basic, intermediate, complex, edge), providing a controlled difficulty landscape for curriculum design and evaluation. Crucially, TAROT decouples curriculum progression from raw reward scores, enabling capability-conditioned evaluation and principled selection from a portfolio of curriculum policies rather than incidental test-case difficulty composition. This design fosters stable optimization and more efficient competency acquisition. Extensive experimental results reveal that the optimal curriculum for RFT in code generation is closely tied to a model's inherent capability, with less capable models achieving greater gains with an easy-to-hard progression, whereas more competent models excel under a hard-first curriculum. TAROT provides a reproducible method that adaptively tailors curriculum design to a model's capability, thereby consistently improving the functional correctness and robustness of the generated code. All code and data are released to foster reproducibility and advance community research at https://github.com/deep-diver/TAROT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑