随着人工智能日益影响日常生活,能耗和数据隐私问题变得尤为突出。在设备上学习直接在边缘设备上训练模型,可以降低能耗并保护数据隐私。然而,现代神经网络的规模不断扩大,给在设备上的训练带来了重大障碍。尽管先前的工作集中在紧凑型卷积架构上,我们转而采用基于子空间的训练方法应用于变压器模型。受模型本质信息位于固定子空间的启发,我们引入了权重-激活子空间迭代(WASI)方法,该方法通过将训练限制在该子空间中,缓解了反向传播的记忆瓶颈,并通过提升变压器模型的推理效率。我们的结果表明,WASI在保持与常规训练相当的准确性的同时,将内存使用量减少了高达62倍,计算成本(FLOPs)减少了高达2倍。在Raspberry Pi 5上,WASI的训练和推理速度比常规训练快约1.4倍。代码可在https://github.com/Le-TrungNguyen/ICLR2026-WASI.git获取。
英文摘要:
As AI increasingly shapes daily life, energy consumption and data privacy have become pressing concerns. On-device learning trains models directly on edge devices, cutting energy consumption and safeguarding data privacy. However, the expanding scale of modern neural networks creates a major obstacle for on-device training. Although prior work has concentrated on compact convolutional architectures, we instead apply subspace-based training to transformer models. Motivated by the idea that a model's essential information lies in a fixed subspace, we introduce Weight-Activation Subspace Iteration (WASI), a method that mitigates the memory bottleneck of backpropagation and boosts inference efficiency in transformer models by restricting training to this subspace. Our results demonstrate that WASI maintains accuracy comparable to vanilla training while reducing memory usage by up to $62\times$ and computational cost (FLOPs) by up to $2\times$. On a Raspberry Pi 5, WASI achieves roughly $1.4\times$ faster training and inference than vanilla training. The code is available at https://github.com/Le-TrungNguyen/ICLR2026-WASI.git.
Attention-Only Transformers via Unrolled Subspace Denoising
Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor(电气工程与计算机科学系,密歇根大学,安阿伯分校);Department of Electrical Engineering and Computer Science, University of California, Berkeley(电气工程与计算机科学系,加州大学伯克利分校);Institute of Data Science & School of Computing and Data Science, University of Hong Kong(数据科学研究所及计算与数据科学学院,香港大学)
Subspace Control: Turning Constrained Model Steering into Controllable Spectral Optimization
OPTML, Michigan State University(OPTML,密歇根州立大学);MIT-IBM Watson AI Lab, IBM Research(MIT-IBM Watson AI Lab,IBM研究院);University of Minnesota(明尼苏达大学);University of Michigan(密歇根大学);University of Colorado Boulder(科罗拉多大学博尔德分校)
Transformer Normalisation Layers and the Independence of Semantic Subspaces
University of Manchester(曼彻斯特大学);Aalto University(阿尔托大学);Idiap Research Institute(Idiap研究所);National Biomarker Centre, CRUK-MI, University of Manchester(英国癌症研究中心曼彻斯特研究所国家生物标志物中心(曼彻斯特大学))
Guessing Efficiently for Constrained Subspace Approximation
University of Utah(犹他大学);Microsoft Research–Redmond(微软研究院-红mond分部);Carnegie Mellon University(卡内基梅隆大学);Toyota Technological Institute at Chicago (TTIC)(丰田技术研究所芝加哥分部)
Simplex, Astera Institute;Sussex Centre for Consciousness Science, University of Sussex;Centre for Complexity Science;Center for Psychedelic Research, Department of Brain Sciences, Imperial College London;Center for Eudaimonia;Human Flourishing, University of Oxford;Beyond Institute for Theoretical Science (BITS)
School of Software Engineering, Xi’an Jiaotong University(西安交通大学软件学院);School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院)
SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRA
Shanghai Qi Zhi Institute(上海启智研究院);Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Xiongan AI Institute(雄安人工智能研究院)
Dimensional Collapse in Transformer Attention Outputs: A Challenge for Sparse Dictionary Learning
Shanghai Innovation Institute, Shanghai, China(上海创新研究院);OpenMOSS Team, School of Computer Science, Fudan University, Shanghai, China(开放MOSS团队,计算机科学学院,复旦大学)
CaTs and DAGs: Integrating Directed Acyclic Graphs with Transformers for Causally Constrained Predictions
Kivira Health(Kivira健康公司);The Sense Innovation and Research Center, CHUV, Lausanne, Switzerland(The Sense创新与研究中心,CHUV,洛桑,瑞士);Centre for Vision, Speech and Signal Processing, University of Surrey, U.K.(视觉、语音和信号处理中心,萨里大学,英国);Institute of Psychology, University of Lausanne, Switzerland(心理学研究所,洛桑大学,瑞士);EPFL, Switzerland(瑞士联邦理工学院)
NASP-T: A Fuzzy Neuro-Symbolic Transformer for Logic-Constrained Aviation Safety Report Classification
Faculty of Science and Technology (REALTEK)(科学与技术学院);Norwegian University of Life Sciences (NMBU)(挪威生命科学大学);Dresden International University (DIU)(德累斯顿国际大学)
Efficient Reinforcement Learning in Factored MDPs with Application to Constrained RL
Key Laboratory of Machine Perception, MOE(教育部机器感知重点实验室);School of EECS, Peking University(北京大学电子工程与计算机科学学院);Amazon(亚马逊公司);Center for Data Science, Peking University(北京大学数据科学中心)
Easz: An Agile Transformer-based Image Compression Framework for Resource-constrained IoTs
Mohamed bin Zayed University of Artificial Intelligence(Mohamed bin Zayed大学人工智能学院);The Hang Seng University of Hong Kong(恒生大学);City University of Hong Kong(城市大学);The Chinese University of Hong Kong(香港中文大学);National Taiwan University(台湾国立台湾大学)