arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.19975cs.LG
收藏

LatentLoop:具有非对称计算分配的缩放循环MoE模型

LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation

SJTU(上海交通大学);Shanghai AI Lab(上海人工智能实验室);USTC(中国科学技术大学);CUHK(香港中文大学);NTU(南洋理工大学)

Zefeng He, Yunkai Chai, Xiaoyang Chen, Tong Zhu, Zongliang Wu, Yun Luo, Yafu Li, Yu Cheng

摘要:

循环混合专家(MoE)模型结合了参数重用和稀疏计算,以提高匹配预算下的性能。以前的工作表明,不同的循环访问可以发挥不同的作用,即使他们共享参数。为了探索不同循环访问的贡献,我们在一个模型中学习单独的残差系数,该模型应用共享块两次,在第一次访问中观察到较小的规模。固定系数的实验表明,减少第一次访问的规模保持或提高性能,而同样的减少到第二伤害它。这种不对称性促使我们探索不对称的计算分配。我们提出了\textbf{LatentLoop},它压缩了第一次访问的序列,在第二次保留完整的分辨率,并重新分配节省的计算,以增加隐藏的大小。我们进一步了解如何强烈的第一次访问更新完整的序列,并共享MoE权重访问,同时使用单独的注意参数,以适应其不同的序列分辨率。在1.3B-7 B参数的15种配置中,最大预算超过$10^{21}$ FLOP,LatentLoop始终降低验证NLL。在拟合的NLL为1.988时,相对于普通MoE和循环基线,预测的计算节省分别为19.0%和7.9%。它还提高了在0杆和5杆设置的所有大小的九个基准的平均得分。这些结果表明,非对称计算分配是一个有前途的方向循环MoE模型。

英文摘要:

Looped Mixture-of-Experts (MoE) models combine parameter reuse with sparse computation to improve performance under matched budgets. Prior work demonstrates that different loop visits can play different roles even when they share parameters. To probe the contributions of different loop visits, we learn separate residual coefficients in a model that applies a shared block twice, observing a smaller scale in the first visit. Fixed coefficient experiments show that reducing the first visit's scale preserves or improves performance, whereas the same reductions to the second hurt it. This asymmetry motivates us to explore asymmetric compute allocation. We propose \textbf{LatentLoop}, which compresses the first visit's sequence, retains full resolution in the second, and reallocates saved computation to increase the hidden size. We further learn how strongly the first visit updates the full sequence, and share MoE weights across visits while using separate attention parameters to accommodate their different sequence resolutions. Across 15 configurations spanning 1.3B--7B parameters, with the largest budget exceeding $10^{21}$ FLOPs, LatentLoop consistently lowers validation NLL. At a fitted NLL of 1.988, predicted compute savings are 19.0\% and 7.9\% relative to vanilla MoE and the looped baseline. It also improves average scores on nine benchmarks at all sizes in 0-shot and 5-shot settings. These results suggest asymmetric compute allocation is a promising direction for looped MoE models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑