arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.23344cs.CLcs.AI
收藏

语言模型学会改进语言模型

Language Models Learn to Improve Language Models

The Pennsylvania State University(宾夕法尼亚州立大学);UC Santa Cruz(加州大学圣克鲁兹分校);Emory University(埃默里大学);UC Santa Barbara(加州大学圣巴巴拉分校);Amazon(亚马逊公司)

Minhua Lin, Zijun Wang, Zewen Liu, Zhaotian Weng, Bing He, Zhan Shi, Benoit Dumoulin, Xiang Zhang, Suhang Wang, Hanqing Lu

AI总结:

本研究提出LM2LM方法,用带RWM奖励模型的RL训练LLM生成LM改进提案,在8种LM开发环境中,该方法让Qwen3.8-27B性能超GPT-6-sol且成本降低33%,还可泛化到未见过的模型家族。

摘要:

大型语言模型(LLMs)正越来越多地被用于自动化AI研究,包括语言模型(LM)开发,为递归自我改进提供了潜在路径。然而,将实验反馈转化为跨模型提出有效改进的学习能力面临两个挑战:其一,用真实反馈训练该能力成本高昂:强化学习(RL)需要评估数万个改进提案,每个提案可能都需要一次完整的模型训练运行,即便每次运行耗时10个GPU小时,仅奖励收集就将消耗数十万个GPU小时,因此需要更廉价的替代方案;其二,当提案分布发生偏移时,策略优化会倾向于高估的提案,导致替代反馈变得不可靠。为应对这些挑战,我们提出LM2LM,该方法用RL训练LLM以生成有效的LM改进提案。为提供可扩展的反馈,我们使用研究世界模型(RWM,一个冻结的LLM,可根据上下文提供的真实实验记录预测提案收益)作为RL的奖励模型。为维持反馈可靠性,我们定期执行RWM排名靠前的当前策略提案,并将其测量结果添加到RWM的上下文中。在8个自回归和扩散LM开发环境中,对保留任务的真实执行评估显示,LM2LM能让Qwen3.8-27B的性能超过GPT-6-sol,即便GPT-6-sol在上下文中获得300条真实实验记录,其数据生成成本仍降低了33%。LM2LM还能泛化到训练期间未见过的模型家族。

英文摘要:

Large language models (LLMs) are increasingly used to automate AI research, including language-model (LM) development, offering a potential path toward recursive self-improvement. However, turning experimental feedback into a learned ability to propose effective improvements across models faces two challenges. First, training such an ability with real feedback is expensive: reinforcement learning (RL) requires evaluating tens of thousands of proposals, each potentially requiring a full model-training run. Even at ten GPU-hours per run, reward collection alone would consume hundreds of thousands of GPU-hours, motivating a cheaper surrogate. Second, surrogate feedback can become unreliable as policy optimization favors overestimated proposals when the proposal distribution shifts. To address these challenges, we propose LM2LM, which trains LLMs with RL to generate effective LM improvement proposals. To provide scalable feedback, we use a research world model (RWM), a frozen LLM that predicts proposal gains from real experimental records provided in context, as the reward model for RL. To maintain feedback reliability, we periodically execute current-policy proposals ranked highly by the RWM and add their measured outcomes to its context. Across eight autoregressive and diffusion LM development environments, real-execution evaluations on held-out tasks show that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even when GPT-6-sol is given 300 real experimental records in context, while requiring 33% lower data-generation cost. LM2LM also generalizes to a model family unseen during training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑