Product of experts (PoE) are layered networks in which the value at each node is an AND (or product) of the values (possibly negated) at its inputs. These were introduced as a neural network architecture that can efficiently learn to generate high-dimensional data which satisfy many low-dimensional constraints -- thereby allowing each individual expert to perform a simple task. PoEs have found a variety of applications in learning.
We study the problem of identifiability of a product of experts model having a layer of binary latent variables, and a layer of binary observables that are iid conditional on the latents. The previous best upper bound on the number of observables needed to identify the model was exponential in the number of parameters. We show: (a) When the latents are uniformly distributed, the model is identifiable with a number of observables equal to the number of parameters (and hence best possible). (b) In the more general case of arbitrarily distributed latents, the model is identifiable for a number of observables that is still linear in the number of parameters (and within a factor of two of best-possible). The proofs rely on root interlacing phenomena for some special three-term recurrences.
Demystifying Softmax Gating Function in Gaussian Mixture of Experts
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Univ. Grenoble Alpes, Inria（格勒诺布尔阿尔卑斯大学，法国国家信息与自动化研究所）；CNRS, Grenoble INP, LJK（法国国家科学研究中心，格勒诺布尔理工学院，让·库尔研究所）
Nonnegative matrix factorizations and related compositional models: Equivalence, identifiability, and an application on the grain-size analysis of sediments
Hangzhou Dianzi University, China（杭州电子科技大学）；Utrecht University, the Netherlands and University of Southampton, UK（乌得勒支大学和南安普顿大学）
CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Tsinghua University（清华大学）
Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps
Faculty of Mathematics and Computer Science, University of Science, Ho Chi Minh City, Vietnam（越南胡志明市科学大学数学与计算机科学学院）；Vietnam National University Ho Chi Minh City, Vietnam（越南胡志明市国家大学）；Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam（越南胡志明市科学大学信息技术学院）；ARC Centre of Excellence for the Mathematical Analysis of Cellular Systems（细胞系统数学分析 excellence 中心）；School of Mathematical Sciences, Queensland University of Technology, Brisbane City, Australia（昆士兰科技大学数学科学学院）；Department of Statistics and Data Science, University of Texas at Austin, Austin, USA（德克萨斯大学奥斯汀分校统计与数据科学系）
Improving Normative Modeling for Multi-modal Neuroimaging Data using mixture-of-product-of-experts variational autoencoders
Department of Computer Science and Engineering, Washington University in St. Louis, USA（计算机科学与工程系，华盛顿大学圣路易斯分校）；Institute for Informatics, Data Science and Biostatistics, Washington University in St.Louis, USA（信息学、数据科学与生物统计研究所，华盛顿大学圣路易斯分校）；Department of Radiology, Washington University in St.Louis, USA（放射学系，华盛顿大学圣路易斯分校）
Language Modeling with Sparse Product of Sememe Experts
Tsinghua University（清华大学）；Tencent（腾讯）
大多数语言建模方法依赖于大规模数据来统计学习单词的顺序模式。在本文中，我们认为，词是原子的语言单位，但不一定是原子的语义单位。受知网的启发，我们使用人类语言中最小的语义单位--义位来表示隐含在词背后的语义，并以此来进行语言建模，我们称之为义位驱动的语言模型（Sememe-Driven Language Model，SDLM）。更具体地说，为了预测下一个单词，SDLM首先估计给定文本上下文的义素分布。然后，它把每个义素看作一个独立的语义专家，这些专家共同识别最可能的义项和对应的词。通过这种方式，SDLM使语言模型能够超越词级操作，实现细粒度的语义级操作，并为我们提供了更强大的工具来微调语言模型，提高语言模型的可解释性和鲁棒性。语言建模和标题生成的下游应用实验表明，SDLM具有显著的效果.实验中使用的源代码和数据可以在https：//www.example.com上访问。github.com/thunlp/SDLM-pytorch
Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Experts
Harvard Medical School and the Brigham and Women’s Hospital（哈佛医学院和布里洛妇产科医院）；MIND Team, Inria Saclay, Université Paris-Saclay, Palaiseau, France and Sorbonne Université, Institut du Cerveau - Paris Brain Institute - ICM, CNRS, Inria, Inserm, AP-HP, Hôpital de la Pitié Salpêtrière（MIND团队、Inria Saclay、巴黎萨克雷大学、Palaiseau，法国以及索邦大学、巴黎脑研究所-ICM、CNRS、Inria、Inserm、AP-HP、皮蒂埃萨勒普蒂耶医院）；Massachusetts Institute of Technology（麻省理工学院）
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Alibaba Group（阿里巴巴集团）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (Shenzhen)（广东人工智能与数字经济实验室（深圳））
Feynman-Kac Correctors in Diffusion: Annealing, Guidance, and Product of Experts
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；McGill University（麦吉尔大学）；Mila - Quebec AI Institute（魁北克AI研究所）；Imperial College London（伦敦帝国学院）；Google DeepMind（谷歌DeepMind）；Duke University（杜克大学）
University of Copenhagen（哥本哈根大学）；University of Montreal（蒙特利尔大学）；Halmstad University（哈尔姆斯塔德大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
A statistical interpretation of spectral embedding: the generalised random dot product graph
University of Bristol（布里斯托大学）；Heilbronn Institute for Mathematical Research（海尔布隆数学研究所）；University of Pittsburgh（匹兹堡大学）；North Carolina State University（北卡罗来纳州立大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
