发表机构
UCI（加州大学欧文分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究符号回归能否发现优于标准优化器的神经网络权重更新规则，通过固定深度符号表达式表示候选规则，在30个组合中25次找到更好规则，总均方误差降低44.47%，表明其可发现紧凑优化器变体但需大规模验证。
AI 中文摘要
我们研究符号回归能否发现明确的神经网络权重更新规则，在小型符号回归基准测试中胜过标准手工设计的优化器。候选更新规则表示为基于常见优化器（包括梯度、动量、自适应梯度和矩估计量）派生的操作数的固定深度符号表达式。在30个基准/神经网络组合中，符号回归程序在25个案例中发现了优于最佳超参数调整的既定优化器的更新规则，改进案例的总均方误差降低了44.47%。发现的规则并非都共享单一的通用符号形式，但许多规则结合了自适应归一化、类似动量的量、非线性变换和有理表达式。这些结果表明，符号回归可作为发现紧凑优化器变体的轻量级机制，同时也凸显了大规模验证的必要性。
英文摘要
We investigate whether symbolic regression can discover explicit neural network weight-update rules that outperform standard hand-designed optimizers on small symbolic regression benchmarks. Candidate update rules are represented as fixed-depth symbolic expressions over operands derived from common optimizers, including gradient, momentum, adaptive-gradient, and moment-estimate quantities. Across 30 benchmark/neural network combinations, the symbolic regression procedure found an update rule outperforming the best hyperparameter-tuned established optimizer in 25 cases, with an aggregate MSE reduction of 44.47\% over the improved cases. The discovered rules do not all share a single common symbolic form, but many combine adaptive normalization, momentum-like quantities, nonlinear transformations, and rational expressions. These results suggest that symbolic regression can serve as a lightweight mechanism for discovering compact optimizer variants, while also highlighting the need for larger-scale validation.
Comments15 pages, 1 figure, 8 tables
