Which Geometry on Which Layer? Measured Sign-Spectral Assignment in Mixed-Optimizer Training
Stanford University(斯坦福大学);Mila(米拉研究所);Université de Montréal(蒙特利尔大学);Arizona State University(亚利桑那州立大学);Nanyang Technological University(南洋理工大学);SB Intuitions Corp.(SB直觉公司)
Hiroki Naganuma, Laura Gomezjurado, Mahdi Ghaznavi, Ganesh Talluri, Atsushi Nitanda, Seng Pei Liew, Ryuichiro Hataya, Ioannis Mitliagkas
Modern neural-network training uses mixed recipes, assigning AdamW-like or sign-based updates to embedding and output layers and spectral updates to hidden matrices. These assignments are usually fixed by layer type, without a clear criterion for selecting the appropriate geometry. We turn sign-versus-spectral assignment into a measurable, layerwise decision problem. For ideal memoryless steps, we derive a finite-batch ratio of conservative descent-lemma lower bounds for spectral and sign updates. Its closed form separates signal efficiency from a geometry-dependent global curvature factor. In the positive-signal regime, the same smoothness assumption gives two-sided bounds that determine which step has the larger conditional expected decrease when the ratio crosses explicit noise-dependent thresholds. We then derive a local directional quadratic score from gradient alignments and Hessian-vector products and use it in Generalized-Scion (G-Scion) to control layerwise routing. In a controlled input-rank experiment, including directional curvature matches the measured same-iterate ordering in all seven aggregate settings, whereas the curvature-free proxy selects the opposite geometry. On 31M GPT-2, the output score routes the output layer from spectral warmup to sign, and G-Scion improves validation loss over fixed Scion in memoryless and matched-momentum comparisons. ViT-B/16 results further show that a spectral edge assignment can outperform fixed Scion outside language modelling. These results suggest that optimizer geometry need not be hard-coded by layer type and are a first step toward understanding how it should be assigned.
Dopamine-driven synaptic credit assignment in neural networks
Indian center for Robotics Innovation(印度机器人创新与智能研究所);Max Planck Institute for Biological Cybernetics(生物 cybernetics 最大计划研究所);Informatics Faculty, Albstadt-Sigmaringen University(信息学院,阿尔布斯塔德-西格马林根大学)
Mixed-signal implementation of feedback-control optimizer for single-layer Spiking Neural Networks
Institute of Neuroinformatics, University of Zürich and ETH Zürich(神经信息学研究所,苏黎世大学和苏黎世联邦理工学院);ETH Zürich, Zürich, Switzerland(苏黎世联邦理工学院,苏黎世,瑞士);AI Center, ETH, Zürich, Switzerland(人工智能中心,苏黎世联邦理工学院,苏黎世,瑞士);Digital Society Initiative, University of Zurich, Switzerland(数字社会倡议,苏黎世大学,瑞士)
MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University(人工智能微电子实验室、人工智能学院、计算机科学学院、上海交通大学);Xiaohongshu Inc.(小红书公司)
Minibatching Offers Improved Generalization Performance for Second Order Optimizers
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Washington State University(华盛顿州立大学);Rutgers, The State University of New Jersey(罗格斯新泽西州立大学);University of Washington(华盛顿大学);Pacific Northwest National Laboratory(太平洋西北国家实验室)
$μ$LO: Compute-Efficient Meta-Generalization of Learned Optimizers
Université de Montréal(蒙特利尔大学);Mila – Quebec AI Institute(魁北克人工智能研究所);Concordia University(康科迪亚大学);Samsung AI Lab(三星AI实验室);ISIR, Sorbonne University, CNRS, Paris, France(巴黎法国索邦大学ISIR研究所、法国国家科学研究中心)
NeuroPlastic: A Plasticity-Modulated Optimizer for Biologically Inspired Learning Dynamics
Department of Neurology, Beth Israel Deaconess Medical Center, Harvard Medical School(神经病学系,贝塞斯达德acons医疗中心,哈佛医学院);Department of Biostatistics, Harvard T.H. Chan School of Public Health(生物统计学系,哈佛T.H. Chan公共卫生学院);Department of Statistics, John A. Paulson School of Engineering and Applied Sciences, Harvard University(统计学系,约翰·A·保罗森工程与应用科学学院,哈佛大学);Department of Psychology, Columbia University(心理学系,哥伦比亚大学)
Sign-Based Optimizers Are Effective Under Heavy-Tailed Noise
State Key Laboratory of Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);School of Software Technology, Zhejiang University(浙江大学软件学院);School of Data Science, Fudan University(复旦大学数据科学学院)
Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy
Fudan University, Shanghai, China(复旦大学);University of Bath, Bath, United Kingdom(巴斯大学);Oxford Suzhou Centre for Advanced Research, Suzhou, China(牛津苏格兰先进研究中心);Shanghai Innovation Institute, Shanghai, China(上海创新研究院);Department of Computer Science, University of Colorado Boulder, Colorado, USA(计算机科学系,科罗拉多大学博尔德分校);Department of Electrical Engineering and Computer Science, University of Michigan(电气工程与计算机科学系,密歇根大学)
GWT: Scalable Optimizer State Compression for Large Language Model Training
National Key Laboratory of Parallel and Distributed Computing(并行与分布式计算国家重点实验室);National University of Defense Technology(国防科技大学);Center for Machine Learning Research(机器学习研究中心)