We analyze a learning-to-optimize (L2O) algorithm that exploits parameter space symmetry to enhance optimization efficiency. Prior work has shown that jointly learning symmetry transformations and local updates improves meta-optimizer performance. Supporting this, our theoretical analysis demonstrates that even without identifying the optimal group element, the method locally resembles Newton's method. We further provide an example where the algorithm provably learns the correct symmetry transformation during training. To empirically evaluate L2O with teleportation, we introduce a benchmark, analyze its success and failure cases, and show that enhancements like momentum further improve performance. Our results highlight the potential of leveraging neural network parameter space symmetry to advance meta-optimization.
Classical symmetries and the Quantum Approximate Optimization Algorithm
Argonne National Laboratory(阿贡国家实验室);NASA Ames Research Center(NASA艾姆斯研究中心);KBR(KBR公司);USRA Research Institute for Advanced Computer Science (RIACS)(美国大学空间研究协会高级计算机科学研究所);University of Delaware(特拉华大学)
Improving the Adaptive Moment Estimation (ADAM) stochastic optimizer through an Implicit-Explicit (IMEX) time-stepping approach
Computational Science Laboratory(计算科学实验室);Virginia Tech(弗吉尼亚理工大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校);California State University, Long Beach(加州州立大学长滩分校)
FastAdaBelief: Improving Convergence Rate for Belief-based Adaptive Optimizers by Exploiting Strong Convexity
University of Science and Technology of China(中国科学技术大学);Duke Kunshan University(昆山杜克大学);Edinburgh Napier University(爱丁堡龙比亚大学);Gusu Laboratory of Materials(姑苏材料实验室)
Department of Physics and Astronomy, University of Exeter, Stocker Road, Exeter EX4 4QL, United Kingdom(物理与天文学系,埃克塞特大学,斯托克罗德路,埃克塞特 EX4 4QL,英国);QuAOS collaboration(QuAOS合作);Institute for Quantum Computing, University of Waterloo, Waterloo, ON N2L 3G1, Canada(量子计算研究所,滑铁卢大学,滑铁卢,ON N2L 3G1,加拿大)
SPRIG: Improving Large Language Model Performance by System Prompt Optimization
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Michigan(密歇根大学);University of Illinois Chicago(伊利诺伊大学芝加哥分校);LG AI Research(LG AI 研究院)
Improving EEG Decoding via Clustering-based Multi-task Feature Learning
Lehigh University(理海大学);Inception Institute of Artificial Intelligence(起源人工智能研究院);South China University of Technology(华南理工大学);Stanford University(斯坦福大学);West China Hospital of Sichuan University(四川大学华西医院);Guangdong University of Technology(广东工业大学);Tokyo University of Agriculture and Technology(东京农工大学);Nicolaus Copernicus University(尼古拉·哥白尼大学)
Learning to optimize with guarantees: a complete characterization of linearly convergent algorithms
School of Electrical Engineering and Computer Science, and Digital Futures, KTH Royal Institute of Technology, Sweden(电气工程与计算机科学学院及数字未来学院,瑞典皇家理工学院);Australian Centre for Robotics and School of Aerospace, Mechanical and Mechatronic Engineering, The University of Sydney, Australia(澳大利亚机器人中心及航空航天、机械与机电工程学院,澳大利亚悉尼大学);Department of Engineering Sciences, University of Oxford, United Kingdom(工程科学系,英国牛津大学)