COMET：利用树和局部搜索学习基数约束的专家混合模型
COMET: Learning Cardinality Constrained Mixture of Experts with Trees and Local Search
AI总结：
本文提出基于树机制的可微稀疏门控 COMET，并引入基于排列的局部搜索以改进稀疏专家选择，在视觉、推荐和 NLP 任务上显著提升性能并降低调参成本。
摘要：
稀疏专家混合（Sparse-MoE）框架在自然语言处理和视觉等多个领域中高效扩展模型容量。Sparse-MoE 使用稀疏的可训练门控为每个输入样本选择一部分“专家”（因此仅选择整个网络的一部分）。现有的稀疏门控在使用一阶优化方法训练时容易出现收敛和性能问题。本文针对当前 MoE 方法提出两项改进。首先，我们提出一种新的稀疏门控：COMET，它依赖一种新颖的基于树的机制。COMET 可微分，能够利用稀疏性加速计算，并且性能优于最先进的门控。其次，由于稀疏专家选择具有颇具挑战性的组合本质，一阶方法通常容易得到低质量解。为应对这一挑战，我们提出一种新颖的、基于排列的局部搜索方法，该方法可在训练任意稀疏门控（例如 Hash routing、Top-k、DSelect-k 和 COMET）时补充一阶方法。我们表明，局部搜索可以帮助网络摆脱糟糕的初始化或解。我们在推荐系统、视觉和自然语言处理等多个领域进行了大规模实验。在标准视觉和推荐系统基准上，COMET+（结合局部搜索的 COMET）相比 Hash routing 和 Top-k 等常用门控在 ROC AUC 上最高实现 13% 的提升，相比 DSelect-k 等先前的可微门控最高实现 9% 的提升。当 Top-k 和 Hash 门控与局部搜索结合时，我们观察到超参数调优所需预算最高减少 100 倍。此外，在语言建模方面，我们的方法在 5/7 个 GLUE 基准以及 SQuAD 数据集上的 BERT 蒸馏任务中优于最先进的 MoEBERT 模型。
英文摘要：
The sparse Mixture-of-Experts (Sparse-MoE) framework efficiently scales up model capacity in various domains, such as natural language processing and vision. Sparse-MoEs select a subset of the "experts" (thus, only a portion of the overall network) for each input sample using a sparse, trainable gate. Existing sparse gates are prone to convergence and performance issues when training with first-order optimization methods. In this paper, we introduce two improvements to current MoE approaches. First, we propose a new sparse gate: COMET, which relies on a novel tree-based mechanism. COMET is differentiable, can exploit sparsity to speed up computation, and outperforms state-of-the-art gates. Second, due to the challenging combinatorial nature of sparse expert selection, first-order methods are typically prone to low-quality solutions. To deal with this challenge, we propose a novel, permutation-based local search method that can complement first-order methods in training any sparse gate, e.g., Hash routing, Top-k, DSelect-k, and COMET. We show that local search can help networks escape bad initializations or solutions. We performed large-scale experiments on various domains, including recommender systems, vision, and natural language processing. On standard vision and recommender systems benchmarks, COMET+ (COMET with local search) achieves up to 13% improvement in ROC AUC over popular gates, e.g., Hash routing and Top-k, and up to 9% over prior differentiable gates e.g., DSelect-k. When Top-k and Hash gates are combined with local search, we see up to $100\times$ reduction in the budget needed for hyperparameter tuning. Moreover, for language modeling, our approach improves over the state-of-the-art MoEBERT model for distilling BERT on 5/7 GLUE benchmarks as well as SQuAD dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
