基于贝叶斯softmax门控专家混合模型
On Bayesian Softmax-Gated Mixture-of-Experts Models
AI总结：
本文研究了贝叶斯专家混合模型的softmax门控机制，分析了后验分布的渐近行为，推导了密度估计、参数估计和模型选择的收敛性保证，并提出了专家数量选择策略。
摘要：
专家混合模型提供了一种灵活的框架，通过输入依赖的门控机制结合多个专家模型，以学习复杂的概率输入输出关系。这些模型在现代机器学习中变得越来越重要，但其在贝叶斯框架下的理论性质仍鲜有研究。本文研究了贝叶斯专家混合模型，重点研究普遍使用的softmax门控机制。具体而言，我们研究了后验分布的渐近行为，针对密度估计、参数估计和模型选择三个基本统计任务。首先，我们建立了密度估计的后验收缩率，包括固定已知专家数量和随机可学习专家数量的两种情形。然后我们分析参数估计并基于定制的Voronoi型损失推导收敛保证，这些损失考虑了专家混合模型复杂的可识别结构。最后，我们提出并分析了两种互补的专家数量选择策略。总体而言，这些结果提供了首个系统性的贝叶斯专家混合模型softmax门控理论分析，并为实际模型设计提供了若干理论指导的见解。
英文摘要：
Mixture-of-experts models provide a flexible framework for learning complex probabilistic input-output relationships by combining multiple expert models through an input-dependent gating mechanism. These models have become increasingly prominent in modern machine learning, yet their theoretical properties in the Bayesian framework remain largely unexplored. In this paper, we study Bayesian mixture-of-experts models, focusing on the ubiquitous softmax-based gating mechanism. Specifically, we investigate the asymptotic behavior of the posterior distribution for three fundamental statistical tasks: density estimation, parameter estimation, and model selection. First, we establish posterior contraction rates for density estimation, both in the regimes with a fixed, known number of experts and with a random learnable number of experts. We then analyze parameter estimation and derive convergence guarantees based on tailored Voronoi-type losses, which account for the complex identifiability structure of mixture-of-experts models. Finally, we propose and analyze two complementary strategies for selecting the number of experts. Taken together, these results provide one of the first systematic theoretical analyses of Bayesian mixture-of-experts models with softmax gating, and yield several theory-grounded insights for practical model design.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
