FlashAttention (Dao, 2023) effectively reduces the quadratic peak memory usage to linear in training transformer-based large language models (LLMs) on a single GPU. In this paper, we introduce DISTFLASHATTN, a distributed memory-efficient attention mechanism optimized for long-context LLMs training. We propose three key techniques: token-level workload balancing, overlapping key-value communication, and a rematerialization-aware gradient checkpointing algorithm. We evaluate DISTFLASHATTN on Llama-7B and variants with sequence lengths from 32K to 512K. DISTFLASHATTN achieves 8x longer sequences, 4.45 - 5.64x speedup compared to Ring Self-Attention, 2 - 8x longer sequences, 1.24 - 2.01x speedup compared to Megatron-LM with FlashAttention. It achieves 1.67x and 1.26 - 1.88x speedup compared to recent Ring Attention and DeepSpeed-Ulysses. Code is available at https://github.com/RulinShao/LightSeq.
Jailbreak-as-a-Service++: Unveiling Distributed AI-Driven Malicious Information Campaigns Powered by LLM Crowdsourcing
Institute of Computing Technology, CAS(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);China People’s Public Security University(中国人民公安大学);Tongji University(同济大学);South China Normal University(华南师范大学);Tsing Hua University(清华大学)
WDMoE: Wireless Distributed Large Language Models with Mixture of Experts
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University(上海交通大学合作媒介网络创新中心);Pengcheng Laboratory(鹏城实验室);Future Network of Intelligent Institute (FNii), the Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳)未来网络智能研究所);Beijing University of Posts and Telecommunications(北京邮电大学)
WDMoE: Wireless Distributed Mixture of Experts for Large Language Models
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University(上海交通大学合作中位数网络创新中心);Pengcheng Laboratory(鹏城实验室);Future Network of Intelligent Institute (FNii), the Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳)未来网络智能研究所);Beijing University of Posts and Telecommunications(北京邮电大学)
AsynDBT: Asynchronous Distributed Bilevel Tuning for efficient In-Context Learning with Large Language Models
Xinjiang Key Laboratory of Intelligent Computing and Smart Applications, School of Software, Xinjiang University(新疆智能计算与智能应用重点实验室,软件学院,新疆大学);Department of Computer Science and Technology, Tongji University(计算机科学与技术系,同济大学);School of Information Science and Engineering, Zaozhuang University(信息科学与工程学院,枣庄大学);Xinjiang University, College of Geography and Remote Sensing Sciences(新疆大学,地理与遥感科学学院);Hochschule Bielefeld-University of Applied Sciences and Arts(比勒菲尔德应用科学与艺术大学);Xinjiang General Station of Exit and Entry Frontier Inspection(新疆出入境边防检查总站)
Edge-Cloud Collaborative Computing on Distributed Intelligence and Model Optimization: A Survey
Division of Natural and Applied Sciences, Duke Kunshan University(杜克-昆山大学自然科学与应用科学系);Department of Electrical and Computer Engineering, The University of British Columbia(不列颠哥伦比亚大学电气与计算机工程系);College of Future Information Technology, Fudan University(复旦大学未来信息科技学院);Ant Group(蚂蚁集团);College of Information Science and Electronic Engineering, Zhejiang University(浙江大学信息科学与电子工程学院)
A Novel Hierarchical Integration Method for Efficient Model Merging in Medical LLMs
Deakin Cyber Research(德肯网络研究);Innovation Centre Deakin University / Universal Higher Education Melbourne, Australia(德肯大学创新中心/通用高等教育墨尔本,澳大利亚);School of IT(信息科技学院);Engineering Melbourne Institute of Technology Melbourne, Australia(工程墨尔本技术学院墨尔本,澳大利亚);Innovation Centre Deakin University Geelong, Australia(德肯大学创新中心吉朗,澳大利亚)
FlowSpec: Continuous Pipelined Speculative Decoding for Efficient Distributed LLM Inference
Department of Computer Science and Engineering, Southern University of Science and Technology(南方科技大学计算机科学与工程系);School of Computing, Montclair State University(蒙特克莱尔州立大学计算机学院);School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院)
HALO: Semantic-Aware Distributed LLM Inference in Lossy Edge Network
Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算机系);Division of Integrative Systems and Design, The Hong Kong University of Science and Technology(香港理工大学系统与设计学院);School of Computer Science and Technology, Huazhong University of Science and Technology(华中科技大学计算机科学与技术学院);Department of Electrical and Computer Engineering, University of Waterloo(滑铁卢大学电气与计算机工程系)