MonkeyTree: 通过迁移实现多租户训练的近最小拥堵
MonkeyTree: Near-Minimal Congestion for Multi-tenant Training via Migration
AI总结：
MonkeyTree通过作业迁移实现多租户GPU集群的近最小网络拥堵，利用机器学习训练流量特性，通过碎片整理优化资源分配，减少作业完成时间。
摘要：
我们提出了MonkeyTree，这是首个通过基于作业迁移的碎片整理来缓解多租户GPU集群中网络拥堵的系统，而不是通过网络层技术。随着云运营商在共享的超订网络上共置机器学习训练作业，拥堵会降低三分之一作业的训练吞吐量。先前的方法要么依赖于路由和流量调度——我们证明当流量超过容量时，这些方法有根本的限制；要么需要昂贵的全双工带宽拓扑结构和包喷射。MonkeyTree利用了机器学习训练流量的特点：基于环的收集操作会为每个作业跨越的机架生成恰好一个跨机架流量，这使得无拥堵的放置成为可能。稀疏的约束结构允许大量的有效配置，使得通过少量迁移即可达到这些配置。一旦达到，低碎片化具有自我强化的特性，因为新到达的作业只会扰动少数机架。MonkeyTree将碎片整理制定为一个整数线性规划问题，该问题在满足每机架碎片限制的情况下最小化工人移动。我们证明了一个紧致的界限，表明任何放置都可以碎片整理到每个ToR最多两个跨机架碎片，并将该公式扩展到混合并行性，每个服务器有多个环。迁移是通过基于RDMA的内存检查点和恢复实现的，在端到端的每个工人上仅产生9.02秒的系统开销。我们使用定制的模拟器评估MonkeyTree，该模拟器建模了最多2048个H200 GPU集群，并在五节点A100测试床上进行原型测试。在1024个GPU的集群中，MonkeyTree比下一个最佳基线提高了14%的平均作业完成时间。在高16:1超订比和2048个GPU的情况下，MonkeyTree将p99作业完成时间保持在理想值的5%以内。
英文摘要：
We present MonkeyTree, the first system to mitigate network congestion in multi-tenant GPU clusters through job-migration based defragmentation rather than network-layer techniques. As cloud operators co-locate ML training jobs on shared, oversubscribed networks, congestion degrades training throughput for over a third of jobs. Prior approaches either rely on routing and flow scheduling--which we show have fundamental limits when traffic exceeds capacity--or require costly full-bisection bandwidth topologies with packet spraying. MonkeyTree exploits characteristics of ML training traffic: ring-based collectives generate exactly one cross-rack flow per rack a job spans, making congestion-free placements achievable. The sparse constraint structure admits abundant valid configurations, making them easy to reach with few migrations. Once reached, low fragmentation is self-reinforcing, as new arrivals disturb only a few racks. MonkeyTree formulates defragmentation as an integer linear program that minimizes worker movements, subject to per-rack fragmentation bounds. We prove a tight bound showing any placement can be defragmented to at most two cross-rack fragments per ToR, and extend the formulation to hybrid parallelism with multiple rings per server. Migration is implemented via in-memory checkpoint-and-restore over RDMA, incurring only 9.02 seconds of system overhead end-to-end per worker. We evaluate MonkeyTree using a custom simulator modeling clusters of up to 2,048 H200 GPUs and prototype on a five-node A100 testbed. MonkeyTree improves average job completion time by 14 percent over the next best baseline on a cluster of 1,024 GPUs with a 4:1 oversubscription. With a high 16:1 oversubscription ratio and 2,048 GPUs, MonkeyTree keeps p99 job completion time within 5 percent of ideal.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
