arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.23951cs.CL
收藏

从AI助手到AI科学家:利用LLM代理自主发现LLM-RL算法

From AI Assistant to AI Scientist: Autonomous Discovery of LLM-RL Algorithms with LLM Agents

Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University(上海数据科学关键实验室,计算机科学学院,复旦大学);School of Data Science, Fudan University(数据科学学院,复旦大学)

Sirui Xia, Yikai Zhang, Aili Chen, Siye Wu, Siyu Yuan, Yanghua Xiao

AI总结:

本文提出POISE框架,通过自动化发现语言模型的策略优化算法,改进了政策优化算法,提高了性能指标。

摘要:

发现改进的语言模型策略优化算法仍是一个成本高昂的手动过程,需要反复的机制级修改和验证。与简单的组合代码搜索不同,这个问题需要在训练动态紧密耦合的算法机制上进行搜索,同时在迭代中重用经验证据。我们提出了POISE,一个闭环框架,用于自动化发现语言模型的策略优化算法。POISE维护一个结构化的、有谱系联系的档案,将提案、可执行实现、标准化评估和自然语言反思联系起来,以支持证据驱动的迭代。在数学推理实验中从GRPO开始,POISE评估了64个候选算法,并发现了改进的机制,包括分析方差缩放和有效性遮蔽。最佳变体将加权Overall从47.8提高到52.5(+4.6),并将AIME25 pass@32从26.7%提高到43.3%,证明了自动化策略优化发现的可行性,同时支持可解释的设计原则。

英文摘要:

Discovering improved policy optimization algorithms for language models remains a costly manual process requiring repeated mechanism-level modification and validation. Unlike simple combinatorial code search, this problem requires searching over algorithmic mechanisms tightly coupled with training dynamics while reusing empirical evidence across iterations. We propose POISE, a closed-loop framework for automated discovery of policy optimization algorithms for language models. POISE maintains a structured, genealogically linked archive linking proposals, executable implementations, standardized evaluations, and natural-language reflections to support evidence-driven iteration. In mathematical reasoning experiments starting from GRPO, POISE evaluates 64 candidate algorithms and discovers improved mechanisms, including analytic-variance scaling and validity masking. The best variant improves weighted Overall from 47.8 to 52.5 (+4.6) and increases AIME25 pass@32 from 26.7% to 43.3%, demonstrating the feasibility of automated policy optimization discovery while supporting interpretable design principles.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑