使用反向传播对大语言模型（LLM）进行微调——即便仅针对LoRA这类参数子集——的内存消耗可能远高于推理，通常被认为在资源受限的移动设备上不具备可行性。零阶优化（ZO）这类替代方法可大幅降低内存占用，但代价是模型收敛速度显著变慢（所需步数是反向传播的10倍到100倍）。我们提出了一种适用于移动设备的内存高效反向传播实现方案（MeBP），可在内存占用与计算时间之间取得更优的权衡，同时收敛速度更快、性能表现优于ZO基线。我们在iPhone 15 Pro Max上验证了MeBP的有效性，结果表明参数量从0.5B到4B不等的各类LLM都可在不足1GB内存的条件下完成微调。我们在https://github.com/apple/ml-mebp 发布了MeBP的实现示例。
英文摘要：
Fine-tuning large language models (LLMs) with backpropagation\textemdash even for a subset of parameters such as LoRA\textemdash can be much more memory-consuming than inference and is often deemed impractical for resource-constrained mobile devices. Alternative methods, such as zeroth-order optimization (ZO), can greatly reduce the memory footprint but come at the cost of significantly slower model convergence (10$\times$ to 100$\times$ more steps than backpropagation). We propose a memory-efficient implementation of backpropagation (MeBP) on mobile devices that provides better trade-off between memory usage and compute time, while converging faster and achieving better performance than the ZO baseline. We verify the effectiveness of MeBP on an iPhone 15 Pro Max and show that various LLMs, ranging from 0.5B to 4B parameters, can be fine-tuned using less than 1GB of memory. We release an example of the MeBP implementation at https://github.com/apple/ml-mebp.
LeanTTA: A Backpropagation-Free and Stateless Approach to Quantized Test-Time Adaptation on Edge Devices
Paul G. Allen School of Computer Science and Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；University of Cambridge（剑桥大学）；School of Computing and Information Systems, University of Melbourne（墨尔本大学计算与信息系统学院）；Samsung AI Center（三星人工智能中心）
Training Long-Context LLMs Efficiently via Chunk-wise Optimization
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室、厦门大学）；OpenGVLab, Shanghai AI Laboratory（上海人工智能实验室）
AdaShadow: Responsive Test-time Model Adaptation in Non-stationary Mobile Environments
Northwestern Polytechnical University（西北工业大学）；City University of Hong Kong（香港城市大学）；The Hong Kong University of Science and Technology（香港科技大学）；Harbin Engineering University（哈尔滨工程大学）
Spiking Neural Networks (SNNs) have recently gained popularity as machine learning models for on-device edge intelligence for applications such as mobile healthcare management and natural language processing due to their low power profile. In such highly personalized use cases, it is important for the model to be able to adapt to the unique features of an individual with only a minimal amount of training data. Meta-learning has been proposed as a way to train models that are geared towards quick adaptation to new tasks. The few existing meta-learning solutions for SNNs operate offline and requ…
Multi-Turn Reasoning LLMs for Task Offloading in Mobile Edge Computing
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Machano-Electronic Engineering, Xidian University（西安电子科技大学机电工程学院）；School of Computer and Communication Engineering, University of Science and Technology Beijing（北京科技大学计算机与通信工程学院）
Zeroth-Order Fine-Tuning of LLMs in Random Subspaces
Beijing Normal University（北京师范大学）；Singapore Management University（新加坡国立大学）；TAL Education Group（TAL教育集团）；Engineering Research Center of Intelligent Technology and Educational Application (MOE)（智能技术与教育应用工程研究中心（教育部））
End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost
University of Georgia（佐治亚大学）；University of North Texas（北卡罗来纳州立大学）；Northeastern University（东北大学）；University of Minnesota（明尼苏达大学）；University of Virginia（弗吉尼亚大学）；Stevens Institute of Technology（史蒂文斯理工学院）
Activation Compression in LLMs: Theoretical Analysis and Efficient Algorithm
Nanjing University（南京大学）；Tsinghua University（清华大学）；Huazhong University of Science and Technology（华中科技大学）；Hong Kong University of Science and Technology（香港科学大学）
Fine-tuning models on edge devices like mobile phones would enable privacy-preserving personalization over sensitive data. However, edge training has historically been limited to relatively small models with simple architectures because training is both memory and energy intensive. We present POET, an algorithm to enable training large neural networks on memory-scarce battery-operated edge devices. POET jointly optimizes the integrated search search spaces of rematerialization and paging, two algorithms to reduce the memory consumption of backpropagation. Given a memory budget and a run-time c…
EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices
Beijing University of Posts and Telecommunications（北京邮电大学）；University of Electronic Science and Technology of China（电子科技大学）；Zhongguangcun Laboratory（中关村实验室）
HKUST-Qualcomm Joint Innovation and Research Laboratory（香港科技大学-高通联合创新研究实验室）；Dept. ECE, Hong Kong University of Science and Technology（香港科技大学电子及计算机工程系）
With the prosperity of mobile devices, the distributed learning approach enabling model training with decentralized data has attracted wide research. However, the lack of training capability for edge devices significantly limits the energy efficiency of distributed learning in real life. This paper describes a novel approach of training DNNs exploiting the redundancy and the weight asymmetry potential of conventional backpropagation. We demonstrate that with negligible classification accuracy loss, the proposed approach outperforms the prior arts by 5x in terms of energy efficiency.
