arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.21072cs.DC
收藏

在互联网规模上通过多维通信优化实现大语言模型的分布式生成推理

Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization

University of California Merced(加州大学默塞德分校);University of California Berkeley(加州大学伯克利分校);College of William & Mary(威廉玛丽学院);Yotta Labs(Yotta实验室)

Jiu Chen, Shuangyan Yang, Xu Xiong, Hexiao Duan, Xinran Zhang, Jie Ren, Dong Li

AI总结:

本文提出BloomBee框架,通过多维优化提升分布式LLM推理性能,降低通信开销,提升吞吐量和减少延迟。

摘要:

去中心化的LLM推理在互联网上异构节点间分配计算,提供高效且成本效益高的解决方案,作为传统集中式推理的替代。然而,节点间网络带宽低使得通信成为主要瓶颈。本文介绍BloomBee,一种互联网规模的分布式LLM推理框架。BloomBee整合了LLM层分配、微批处理和张量卸载,从多个维度优化通信。此外,BloomBee将这些技术的协调制定为优化问题,并通过动态规划求解。BloomBee还根据低带宽网络设置定制无损压缩和推测解码,以减少通信开销。我们在各种网络环境中评估了BloomBee,显示其在吞吐量上提升了最高1.76倍,相比最先进的去中心化LLM推理系统,平均延迟降低了最高43.20%。BloomBee已开源。

英文摘要:

Decentralized LLM inference distributes computation among heterogeneous nodes across the internet, offering a performant and cost-efficient solution, alternative to traditional centralized inference. However, the low cross-node network bandwidth makes communication the primary bottleneck. In this paper, we introduce BloomBee, an internet-scale distributed LLM inference framework. BloomBee integrates LLM-layer assignment, micro-batching and tensor offloading to optimize communication from multiple dimensions. Additionally, BloomBee formulates the coordination of these techniques as an optimization problem and solves it using dynamic programming. BloomBee also customizes lossless compression and speculative decoding according to low-bandwidth network settings to reduce communication overhead. We evaluate BloomBee across a spectrum of network environments and show that it improves service throughput by up to 1.76x. It also reduces average latency by up to 43.20% compared to state-of-the-art decentralized LLM inference systems. BloomBee is open-sourced.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑