Recently, various distributed strategies for large language model training have been proposed. However, these methods provided limited solutions for the trade-off between memory consumption and communication cost. In this paper, we rethink the impact of memory consumption and communication costs on the training speed of large language models, and propose a memory-communication balanced strategy set Partial Redundancy Optimizer (PaRO). PaRO provides comprehensive options which reduces the amount and frequency of inter-group communication with minor memory redundancy by fine-grained sharding strategy, thereby improving the training efficiency in various training scenarios. Additionally, we propose a Hierarchical Overlapping Ring (HO-Ring) communication topology to enhance communication efficiency between nodes or across switches in large language model training. Our experiments demonstrate that PaRO significantly improves training throughput by 1.19x-2.50x compared to the SOTA method and achieves a near-linear scalability. The HO-Ring algorithm improves communication efficiency by 36.5% compared to the traditional Ring algorithm.
PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs
Department for AI in Society, Science, and Technology, Zuse Institute Berlin, Germany(人工智能与社会、科学与技术部门,柏林Zuse研究所);Institute of Mathematics, Technische Universität Berlin, Germany(数学研究所,柏林技术大学)
Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy
Fudan University, Shanghai, China(复旦大学);University of Bath, Bath, United Kingdom(巴斯大学);Oxford Suzhou Centre for Advanced Research, Suzhou, China(牛津苏格兰先进研究中心);Shanghai Innovation Institute, Shanghai, China(上海创新研究院);Department of Computer Science, University of Colorado Boulder, Colorado, USA(计算机科学系,科罗拉多大学博尔德分校);Department of Electrical Engineering and Computer Science, University of Michigan(电气工程与计算机科学系,密歇根大学)
Rethinking Efficient Graph Coarsening via a Non-Selfishness Principle
School of Information Science and Electronic Engineering, Shanghai Jiao Tong University, Shanghai, China(上海交通大学信息科学与电子工程学院);School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China(上海交通大学人工智能学院);School of Environment Science and Engineering, Shanghai Jiao Tong University, Shanghai, China(上海交通大学环境科学与工程学院);School of Artificial Intelligence, Nanchang University, Nanchang, China(南昌大学人工智能学院);Institute of Geographic Sciences and Natural Resources Research, Chinese Academy of Sciences, Beijing, China(中国科学院地理科学与资源研究所)
Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective
College of Computer Science, Sichuan University, Chengdu, China(四川大学计算机学院,成都,中国);Institute of High Performance Computing, A STAR, Singapore(高性能计算研究院,新加坡)
Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximation
University of Science and Technology of China(中国科学技术大学);NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Rethinking Federated Graph Learning: A Data Condensation Perspective
Computer Network Information Center, Chinese Academy of Sciences(中国科学院计算机网络信息中心);University of Chinese Academy of Sciences(中国科学院大学);Beijing Institute of Technology(北京理工大学);Sun Yat-sen University(中山大学)
Rethinking Efficiency in Neural Combinatorial Optimization: Batched Preference Optimization with Mamba
National Key Laboratory of Big Data and Decision, National University of Defense Technology(大数据与决策国家重点实验室,国防科技大学);School of Computer Science, South China University of Technology(华南理工大学计算机科学学院);School of Computing and Information Systems, Singapore Management University(新加坡管理学院计算机与信息系统学院)
Rethinking Federated Learning Over the Air: The Blessing of Scaling Up
Zhejiang University/University of Illinois Urbana-Champaign Institute(浙江大学/伊利诺伊大学厄巴纳-香槟分校研究所);Zhejiang University(浙江大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Engineering Systems and Design, Singapore University of Technology and Design(新加坡科技与设计大学工程系统与设计系);School of Computer Science, Nanjing University of Posts and Telecommunications(南京邮电大学计算机科学学院);Department of Computer and Information Science, Linköping University(_linköping大学计算机与信息科学系)