We present a method to formulate algorithm discovery as program search, and apply it to discover optimization algorithms for deep neural network training. We leverage efficient search techniques to explore an infinite and sparse program space. To bridge the large generalization gap between proxy and target tasks, we also introduce program selection and simplification strategies. Our method discovers a simple and effective optimization algorithm, $\textbf{Lion}$ ($\textit{Evo$\textbf{L}$ved S$\textbf{i}$gn M$\textbf{o}$me$\textbf{n}$tum}$). It is more memory-efficient than Adam as it only keeps track of the momentum. Different from adaptive optimizers, its update has the same magnitude for each parameter calculated through the sign operation. We compare Lion with widely used optimizers, such as Adam and Adafactor, for training a variety of models on different tasks. On image classification, Lion boosts the accuracy of ViT by up to 2% on ImageNet and saves up to 5x the pre-training compute on JFT. On vision-language contrastive learning, we achieve 88.3% $\textit{zero-shot}$ and 91.1% $\textit{fine-tuning}$ accuracy on ImageNet, surpassing the previous best results by 2% and 0.1%, respectively. On diffusion models, Lion outperforms Adam by achieving a better FID score and reducing the training compute by up to 2.3x. For autoregressive, masked language modeling, and fine-tuning, Lion exhibits a similar or better performance compared to Adam. Our analysis of Lion reveals that its performance gain grows with the training batch size. It also requires a smaller learning rate than Adam due to the larger norm of the update produced by the sign function. Additionally, we examine the limitations of Lion and identify scenarios where its improvements are small or not statistically significant. Lion is also successfully deployed in production systems such as Google search ads CTR model.
Symbolic Graph Networks for Robust PDE Discovery from Noisy Sparse Data
School of Software Engineering, Chengdu University of Information Technology(成都信息科技学院软件工程学院);School of Statistics, Chengdu University of Information Technology(成都信息科技学院统计学院);College of Applied Mathematics, Chengdu University of Information Technology(成都信息科技学院应用数学学院);Key Laboratory of Mathematical Meteorology, Chengdu University of Information Technology(成都信息科技学院数学气象重点实验室)
Self-Evolving Multi-Agent Symbolic Discovery for Financial Fundamental Analysis
National University of Singapore(新加坡国立大学);Asian Institute of Digital Finance(亚洲数字金融研究院);KTH Royal Institute of Technology(KTH皇家理工学院);University of Oxford(牛津大学)
LieDiscover: Adaptive Symbolic Library Construction for Explicit Open-form Symmetry Discovery
East China Normal University(华东师范大学);Shanghai Jiao Tong University(上海交通大学);Beihang University(北京航空航天大学);Institute of Applied Physics and Computational Mathematics(应用物理与计算数学研究所);Northeast Normal University(东北师范大学)
Bias Inheritance in Neural-Symbolic Discovery of Constitutive Closures Under Function-Class Mismatch
Nanophotonics and Biophotonics Key Laboratory of Jilin Province, School of Physics, Changchun University of Science and Technology(吉林省纳米光子学与生物光子学重点实验室,物理学院,长春理工大学)
Automated discovery of symbolic laws governing skill acquisition from naturally occurring data
National Engineering Research Center of Educational Big Data, Central China Normal University(华中师范大学教育大数据国家工程研究中心);Faculty of Artificial Intelligence in Education, Central China Normal University(华中师范大学人工智能教育学部);National Engineering Research Center for E-learning, Central China Normal University(华中师范大学国家数字化学习工程技术研究中心)
Neuro-Symbolic Skill Discovery for Conditional Multi-Level Planning
Department of Computer Science and Technology The University of Cambridge(计算机科学与技术系 剑桥大学);Department of Computer Engineering Bogazici University(计算机工程系 boazici大学);SISREC Osaka University(Osaka大学SISREC)
Data-Efficient Symbolic Regression via Foundation Model Distillation
Institute for Clarity in Documentation(文档清晰研究所);Inria Paris-Rocquencourt(巴黎-罗克琴特研究所);Rajiv Gandhi University(拉吉夫·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕尔默研究实验室);Arizona State University(亚利桑那州立大学);Clemson University(克莱姆森大学)
Symbolic Neural Generation with Applications to Lead Discovery in Drug Design
Dept. of Computer Science & Information Systems and APPCAIR BITS Pilani, K K Birla Goa Campus, India(计算机科学与信息系统系及APPCAIR比特纳学院,K K Birl拉果阿校区,印度);Dept. of Computer Science & Information Systems BITS Pilani, K K Birla Goa Campus, India(计算机科学与信息系统系比特纳学院,K K Birl拉果阿校区,印度);Department of Biochemistry, University of Cambridge, Cambridge, UK(生物化学系,剑桥大学,剑桥,英国);School of Computer Science and Engineering University of New South Wales, Sydney(计算机科学与工程学院新南威尔士大学,悉尼);Dr. B.R. Ambedkar Center for Biomedical Research University of Delhi, New Delhi, India(B.R.阿姆贝卡尔生物医学研究中心,德里大学,新德里,印度);Department of Chemistry BITS Pilani, K.K. Birla Goa Campus, India(化学系比特纳学院,K.K. Birl拉果阿校区,印度)
T-SHRED: Symbolic Regression for Regularization and Model Discovery with Transformer Shallow Recurrent Decoders
Electrical and Computer Engineering, University of Washington(华盛顿大学电气与计算机工程系);Applied Mathematics, University of Washington(华盛顿大学应用数学系);Computer Science & Engineering, University of Washington(华盛顿大学计算机科学与工程系)
DeepPySR -- A Symbolic Regression Framework with Dynamic Pruning, Pareto Selection, and Hierarchical Composition for Real-World Scientific Discovery
International Centre for Radio Astronomy Research (ICRAR), University of Western Australia(国际射电天文学研究中心(ICRAR),西澳大利亚大学);The Raine Study, University of Western Australia(雷恩研究中心,西澳大利亚大学);Menzies Institute for Medical Research, University of Tasmania(孟席斯医学研究所,塔斯马尼亚大学);Nutrition & Health Innovation Research Institute, School of Medical and Health Sciences, Edith Cowan University(营养与健康创新研究所,伊迪斯科文大学医学与健康科学学院)
AutoSINDy: Discovery of Nonlinear Dynamics with Automated Library Construction
Data Science and Analytics Institute, University of Oklahoma, Norman, OK, USA(数据科学与分析研究所,俄克拉荷马大学,诺曼,OK,USA);School of Industrial and Systems Engineering, University of Oklahoma, Norman, OK, USA(工业与系统工程学院,俄克拉荷马大学,诺曼,OK,USA)
Any equation is a forest: Symbolic genetic algorithm for discovering open-form partial differential equations (SGA-PDE)
Peng Cheng Laboratory(鹏城实验室);University of Washington(华盛顿大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Peking University(北京大学);Southern University of Science and Technology(南方科技大学)
Structured Kolmogorov-Arnold Neural ODEs for Interpretable Learning and Symbolic Discovery of Nonlinear Dynamics
Department of Industrial Systems Engineering and Management, National University of Singapore(工业系统工程与管理系,新加坡国立大学);Department of Civil, Environmental and Geomatic Engineering, ETH Zürich(土木、环境与测绘工程系,苏黎世联邦理工学院);Future Resilient Systems, Singapore-ETH Centre(未来韧性系统,新加坡-苏黎世中心)