TerraceMoE：一种用于分层MoE全对全通信的成本模型
TerraceMoE: A Cost Model for Hierarchical MoE All-to-All Communication
AI总结：
该研究针对分层MoE全对全通信提出TerraceMoE成本模型，通过该模型得出不同场景下的盈亏平衡分层比率等结果，为MoE训练的通信权衡提供了量化依据。
AI中文摘要：
分层两跳调度可减少专家并行混合专家（Mixture-of-Experts，MoE）训练中的慢结构流量，但会增加第二个集合操作和到达侧算子链。我们提出一种用于在通信调用级别权衡筛选的成本模型，该模型受验证门限约束，当门限不满足时会在代码中撤回能力，而非报告警告。在参考几何配置（16个组，每组8个秩，q=3，H=2048，每个秩4096个token）下，测量的PyTorch到达链对应的修正有效盈亏平衡分层比率为3.98，假设的融合目标对应为1.49，零实现开销时为1.10。这些仅为比率敏感性结果，而非部署预测：平台A测量值为1.03，平台B无分离的快/慢测量，两台机器均未达到分层 regime。四个通信级语料库通过其门限；一个漂移探测和步级门限失败，后者在代码中强制执行，因此我们不做训练吞吐量预测。启用的路由约束固定每个token的扇出和每个选定组的配额，而每个对等节点的总计数仍依赖数据。其测量的验证损失成本很小但非零（+0.0034 nats）；下游等效性报告的估计器来源不完整，因此无法从产物中独立重构。代码、校准常数和验证门限位于此https URL。
英文摘要：
Hierarchical two-hop dispatch can reduce slow-fabric traffic in expert-parallel Mixture-of-Experts training, but it adds a second collective and an arrival-side operator chain. We present a cost model for screening that trade at the communication-call level, bounded by validation gates that withdraw a capability in code when they fail rather than reporting a caveat. At a reference geometry with 16 groups of 8 ranks, $q=3$, $H=2048$, and 4096 tokens per rank, the corrected effective breakeven hierarchy ratio is 3.98 for the measured PyTorch arrival chain, 1.49 for a hypothetical fused target, and 1.10 at zero implementation overhead. These are ratio-only sensitivity results, not deployment predictions: platform A measures 1.03, platform B has no separated fast/slow measurement, and neither machine measured here reaches the hierarchical regime. Four communication-level corpora pass their gates; a drift probe and the step-level gate fail. The latter failure is enforced in code, so we make no training-throughput prediction. The enabling routing constraint fixes per-token fan-out and per-selected-group quota, while aggregate per-peer counts remain data-dependent. Its measured validation-loss cost is small but nonzero (+0.0034 nats); downstream equivalence is reported with incomplete estimator provenance and is therefore not independently reconstructible from the artifact. Code, calibration constants and the validation gates are at https://github.com/weich97/TerraceMoE-simulator.
