arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.20835cs.CL
收藏

并行SFT:改进代码RL的零样本跨编程语言转移

Parallel-SFT: Improving Zero-Shot Cross-Programming-Language Transfer for Code RL

Meta Superintelligence Labs(Meta超智能实验室);MIT(麻省理工学院)

Zhaofeng Wu, Shiqi Wang, Boya Peng, Anuj Goyal, Melanie Kambadur, Sebastian Ruder, Yoon Kim, Chloe Bi

AI总结:

本文提出零样本跨编程语言转移任务,通过并行SFT策略提升代码RL在不同语言间的迁移能力,发现通用化SFT初始化能提升模型泛化性能。

摘要:

现代语言模型在常见编程语言(如C++和Python)中表现出色,但在低资源编程语言中的表现受限于训练数据。然而,大多数编程技能在不同语言间应具有通用性,因此在一种语言中获得的能力应能迁移到其他语言。本文提出零样本跨编程语言转移任务,发现Llama-3.1在源语言中进行RL训练时,对目标语言性能无改善甚至退化。为此,我们假设有效的RL迁移需要在RL前进行可泛化的SFT初始化。为此,我们提出并行SFT策略,将功能等价的多语言代码纳入数据混合。我们证明此方法提升了迁移能力:在后续对并行SFT模型进行RL训练时,观察到对未见过的语言有更好的泛化能力。模型内部表示分析显示,并行SFT导致更以功能为中心的潜在空间,其中不同语言中的等价程序更紧密聚类,这可能有助于提升迁移能力。

英文摘要:

Modern language models demonstrate impressive coding capabilities in common programming languages (PLs), such as C++ and Python, but their performance in lower-resource PLs is often limited by training data availability. In principle, however, most programming skills are universal across PLs, so the capability acquired in one PL should transfer to others. In this work, we propose the task of zero-shot cross-programming-language transfer for code RL. We find that, for Llama-3.1, RL training for code generation in a source PL fails to improve, and sometimes even degrades, the performance on other target PLs. To address this, we hypothesize that effective RL transfer requires a generalizable SFT initialization before RL. We thus propose **Parallel-SFT**, an SFT strategy that incorporates "parallel programs" -- functionally equivalent code implemented in multiple PLs -- into the data mixture. We demonstrate that this improves transferability: when we subsequently perform RL on our Parallel-SFT model, we observe better generalization to unseen PLs. Analysis of the model internal representations reveals that Parallel-SFT leads to a more functionality-centric latent space, where equivalent programs across PLs are more tightly clustered, which we hypothesize to contribute to the improved transferability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑