Google Research（谷歌研究院）；University of Cambridge（剑桥大学）；Waymo；ETH Zürich（苏黎世联邦理工学院）
James Urquhart Allingham, Florian Wenzel, Zelda E Mariet, Basil Mustafa, Joan Puigcerver, Neil Houlsby, Ghassen Jerfel, Vincent Fortuin, Balaji Lakshminarayanan… 展开作者
James Urquhart Allingham, Florian Wenzel, Zelda E Mariet, Basil Mustafa, Joan Puigcerver, Neil Houlsby, Ghassen Jerfel, Vincent Fortuin, Balaji Lakshminarayanan, Jasper Snoek, Dustin Tran, Carlos Riquelme Ruiz, Rodolphe Jenatton
Machine learning models based on the aggregated outputs of submodels, either at the activation or prediction levels, often exhibit strong performance compared to individual models. We study the interplay of two popular classes of such models: ensembles of neural networks and sparse mixture of experts (sparse MoEs). First, we show that the two approaches have complementary features whose combination is beneficial. This includes a comprehensive evaluation of sparse MoEs in uncertainty related benchmarks. Then, we present Efficient Ensemble of Experts (E$^3$), a scalable and simple ensemble of sparse MoEs that takes the best of both classes of models, while using up to 45% fewer FLOPs than a deep ensemble. Extensive experiments demonstrate the accuracy, log-likelihood, few-shot learning, robustness, and uncertainty improvements of E$^3$ over several challenging vision Transformer-based baselines. E$^3$ not only preserves its efficiency while scaling to models with up to 2.7B parameters, but also provides better predictive performance and uncertainty estimates for larger models.
Mixture of ELM based experts with trainable gating network
Yazd University（亚兹德大学）
混合专家模型方法是一种基于神经网络的集成学习方法，具有很强的提高整体分类精度的能力。该方法基于分而治之的原则，通过选通网络的监督将问题空间划分给多个专家。本文提出了一种基于混合专家模型的集成学习方法--MEETG（mixed of ELM based experts with trainable gating networks），它由多层感知器（MLP）和门网络组成，其中应用基于梯度的学习算法来训练MLP，这是一个迭代且耗时的过程。为了克服这些问题，我们利用极限学习机（ELM）的优点设计了ME的结构，ELM作为单隐层前馈神经网络的学习算法，与其他传统的学习算法相比，具有更快的学习速度和更好的泛化能力。此外，在所提出的方法中，一个可训练的门控网络被应用于根据输入样本动态地聚集专家的输出。在11个基准数据集上的实验结果和统计分析证实了MEETG在分类问题上具有可接受的性能。此外，我们的实验结果表明，该方法在预测稳定性和分类准确性方面优于原始ELM。
Deep sub-ensembles meets quantile regression: uncertainty-aware imputation for time series
Department of Computer and Information（计算机与信息系）；Hefei University of Technology（合肥工业大学）；Dept. of Comp. Sci. & Tech.（计算机科学与技术系）；Tsinghua University（清华大学）
Ensemble Kalman Filtering Meets Gaussian Process SSM for Non-Mean-Field and Online Inference
School of Science & Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）理工学院）；Future Network of Intelligence Institute（未来智能网络研究院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；National University of Singapore（新加坡国立大学）
