发表机构
Institute of Automation, Chinese Academy of Sciences; University of Chinese Academy of Sciences; Zhongguancun Academy（中国科学院自动化研究所; 中国科学院大学; 中关村科学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究针对深度学习优化器设计难题，提出OPTScientist多智能体框架，在类型化DSL中结合进化搜索与第二阶段机制，发现RS-MR优化器改进Transformer预训练，为自动优化器科学提供新路径。
AI 中文摘要
为现代深度学习设计优化器仍然是一个具有挑战性的科学问题，需要综合考虑优化几何、状态动态、数值稳定性、实现约束和经验泛化。现有自动优化器发现方法要么在无约束代码空间中搜索，要么在参数化狭窄的优化器家族中搜索。前者灵活但常产生无效或不可解释的程序，后者稳定但限制新颖性。我们引入OPTScientist，一种在类型化领域特定语言（DSL）中用于优化器发现的理论引导多智能体框架。它将优化器设计表述为受约束的科学搜索过程，四个角色智能体协作提出假设、合成DSL候选、编译和评估优化器并评判结果。为克服固定搜索空间的限制，它结合了对优化器程序的进化搜索和第二阶段机制。使用该框架，我们发现了RS-MR，一种在我们的原生评估协议下比强大基线更能改进Transformer预训练的简化状态矩阵优化器。我们的结果为基于理论、类型化程序、编译器验证和闭环实验的自动优化器科学指明了道路。
英文摘要
Designing optimizers for modern deep learning remains a challenging scientific problem, requiring the joint consideration of optimization geometry, state dynamics, numerical stability, implementation constraints, and empirical generalization. Existing automated optimizer discovery methods typically search either over unconstrained code spaces or within narrowly parameterized optimizer families. The former is flexible but often produces invalid or uninterpretable programs, while the latter is stable but limits novelty. We introduce OPTScientist, a theory-guided multi-agent framework for optimizer discovery in a typed domain-specific language (DSL). OPTScientist formulates optimizer design as a constrained scientific search process, where candidate updates are expressed through direction, scaling, preconditioning, regularization, state, and grouping modules. Four role agents, Theorist, Designer, Engineer, and Reviewer, collaborate within a single orchestration loop to propose hypotheses, synthesize DSL candidates, compile and evaluate optimizers, and critique results. To overcome the limitations of a fixed search space, OPTScientist combines evolutionary search over optimizer programs with a second-stage mechanism that proposes small DSL extensions when repeated failures reveal representational bottlenecks. Using this framework, we discover RS-MR, a reduced-state matrix optimizer that improves transformer pretraining over strong baselines under our native evaluation protocol. Our results suggest a path toward automated optimizer science grounded in theory, typed programs, compiler validation, and closed-loop experimentation.
