带自适应路由的专家软合并（Soft Merging of Experts with Adaptive Routing）
Soft Merging of Experts with Adaptive Routing
AI总结：
针对稀疏激活模型中离散路由导致的训练性能不足问题，提出通过加权平均专家参数构建合并专家的SMEAR方法，在不显著增加计算量的前提下实现可微训练，性能优于多种基线路由方法且专家具备专业化特征。
摘要：
采用条件计算的稀疏激活神经网络会学习将输入路由到不同的“专家”子网络，提供了一种密集激活模型所不具备的模块化形式。尽管这类模型有潜在优势，但采用学习式路由的模型性能往往不如参数规模匹配的密集激活模型，也不及使用非学习式启发式路由策略的模型。\n在本文中，我们假设这些不足源于训练采用不可微离散路由决策的稀疏激活模型时所使用的梯度估计技术。为解决这一问题，我们提出了带自适应路由的专家软合并方法（Soft Merging of Experts with Adaptive Routing, SMEAR），该方法通过对所有专家参数进行加权平均构建出单个“合并”专家，从而避免了离散路由。通过将激活值输入单个合并专家进行路由，SMEAR不会显著增加计算成本，且支持标准的基于梯度的训练。\n我们通过实验验证，采用SMEAR的模型性能优于基于元数据路由的模型，也优于通过梯度估计学习稀疏路由的模型。此外，我们提供的定性分析表明，通过SMEAR学习到的专家呈现出显著的专业化特征。实验中使用的所有代码均已公开。
英文摘要：
Sparsely activated neural networks with conditional computation learn to route their inputs through different "expert" subnetworks, providing a form of modularity that densely activated models lack. Despite their possible benefits, models with learned routing often underperform their parameter-matched densely activated counterparts as well as models that use non-learned heuristic routing strategies. In this paper, we hypothesize that these shortcomings stem from the gradient estimation techniques used to train sparsely activated models that use non-differentiable discrete routing decisions. To address this issue, we introduce Soft Merging of Experts with Adaptive Routing (SMEAR), which avoids discrete routing by using a single "merged" expert constructed via a weighted average of all of the experts' parameters. By routing activations through a single merged expert, SMEAR does not incur a significant increase in computational costs and enables standard gradient-based training. We empirically validate that models using SMEAR outperform models that route based on metadata or learn sparse routing through gradient estimation. Furthermore, we provide qualitative analysis demonstrating that the experts learned via SMEAR exhibit a significant amount of specialization. All of the code used in our experiments is publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
