本文改进神经符号 Differentiable Tree Machine,利用 mixture of experts 使参数量不随计算步数增长,并提出无 oracle 的终止算法,让模型自动学习并收敛到最优步数。
摘要:
我们推进了最近提出的神经符号 Differentiable Tree Machine(可微树机器),该模型结合 transformers 与 Tensor Product Representations(张量积表示)来学习树操作。我们研究了其架构,并提出两个关键组件。首先,我们通过引入 mixture of experts(专家混合),移除了在每一步中都会使用的一系列不同 transformer 层。由此得到的 Differentiable Tree Experts(可微树专家)模型,在计算中无论步数多少都具有恒定数量的参数;相比之下,此前 Differentiable Tree Machine 中的方法会随步数线性增长。鉴于步数方面的这种灵活性,我们还提出了一种新的终止算法,使模型能够自动选择执行多少步。所得 Terminating Differentiable Tree Experts(终止型可微树专家)模型在没有 oracle 的情况下,能够缓慢地学习预测步数。它可以在保持模型学习能力的同时做到这一点,并收敛到最优步数。
英文摘要:
We advance the recently proposed neuro-symbolic Differentiable Tree Machine, which learns tree operations using a combination of transformers and Tensor Product Representations. We investigate the architecture and propose two key components. We first remove a series of different transformer layers that are used in every step by introducing a mixture of experts. This results in a Differentiable Tree Experts model with a constant number of parameters for any arbitrary number of steps in the computation, compared to the previous method in the Differentiable Tree Machine with a linear growth. Given this flexibility in the number of steps, we additionally propose a new termination algorithm to provide the model the power to choose how many steps to make automatically. The resulting Terminating Differentiable Tree Experts model sluggishly learns to predict the number of steps without an oracle. It can do so while maintaining the learning capabilities of the model, converging to the optimal amount of steps.
SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs
AGH University of Krakow, Poland(克拉科夫AGH大学);Faculty of Mathematics(数学系);Computer Science, Jagiellonian University, Poland(计算机科学系,杰哥利安大学,波兰);Centre for Credible Artificial Intelligence, Warsaw University of Technology(可信人工智能中心,华沙技术大学)
Densities of Almost Surely Terminating Probabilistic Programs are Differentiable Almost Everywhere
University of Oxford(牛津大学)
研究了具有递归和条件的高阶统计概率规划的微分性质。我们的出发点是Hongseok Yang提出的一个开放问题:什么样的统计概率程序具有几乎处处可微的密度?为了正式的问题,我们考虑统计PCF(SPCF),呼叫值PCF与实数的扩展,和结构的采样和调节。我们给SPCF一个采样风格的操作语义a la Borgstrom et al.,并研究可能的执行轨迹集合上的相关联的权重(通常称为密度)函数和价值函数。我们的主要结果是,几乎肯定终止的SPCF程序,从一组原始函数(例如一组解析函数)满足温和的封闭性,具有重量和价值函数,几乎处处可微。我们使用符号执行的随机形式来推理几乎处处可微性。这项工作的一个副产品是,几乎肯定终止的确定性(S)PCF程序与实参数表示的功能,几乎处处可微。我们的结果是有实际意义的,因为几乎无处不在的密度函数的可微性,主要的基于梯度的推理算法的正确性。
Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization
Queen Mary University of London(伦敦玛丽女王大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);City University of London(伦敦城市大学);National and Kapodistrian University of Athens(雅典国立卡波迪斯特里亚大学);Archimedes AI, Athena RC(阿基米德人工智能公司,雅典娜研究中心);The Cyprus Institute(塞浦路斯研究所);Imperial College London(帝国理工学院)
Differentiable Quantum Architecture Search in Quantum-Enhanced Neural Network Parameter Generation
Wells Fargo(摩根大通);Graduate Institute of Applied Physics(应用物理研究所);Imperial College London(帝国理工学院);Centre for Quantum Engineering, Science and Technology (QuEST)(量子工程、科学与技术中心);Hon Hai (Foxconn) Research Institute(鸿海(富士康)研究院);Quantum Information Center(量子信息中心);Master Program in Intelligent Computing and Big Data(智能计算与大数据硕士项目);Jij inc.(Jij公司)
ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning
Department of Computer Science, Florida State University(佛罗里达州立大学计算机科学系);Department of Computer Science and Engineering, University of Notre Dame(圣母大学计算机科学与工程系);Department of Computer Science, University of Maryland, College Park(马里兰大学学院市计算机科学系);Samsung Research America(三星美国研究机构);Department of Computer Science, University of Central Florida(中央佛罗里达大学计算机科学系);Department of Health Outcomes and Biomedical Informatics, University of Florida(佛罗里达大学健康结果与生物医学信息学系)
NASP-T: A Fuzzy Neuro-Symbolic Transformer for Logic-Constrained Aviation Safety Report Classification
Faculty of Science and Technology (REALTEK)(科学与技术学院);Norwegian University of Life Sciences (NMBU)(挪威生命科学大学);Dresden International University (DIU)(德累斯顿国际大学)
NAS-FM: Neural Architecture Search for Tunable and Interpretable Sound Synthesis based on Frequency Modulation
Hong Kong Baptist University(香港浸会大学);Hong Kong University of Science and Technology(香港科技大学);Microsoft Research Asia(微软亚洲研究院);Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences(中国科学院香港创新研究院)
DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physics
University of California, San Diego(加州大学圣地亚哥分校);Tsinghua University(清华大学);Autodesk(欧特克);UMass Amherst(马萨诸塞大学阿默斯特分校);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室)
Estimate, Don't Imitate: Reusing Differentiable State-Based Policies for Visuomotor Control
Fraunhofer Heinrich-Hertz-Institut(弗劳恩霍夫海因里希·赫兹研究所);Technische Universität Berlin(柏林工业大学);BIFOLD – Berlin Institute for the Foundations of Learning and Data(BIFOLD – 柏林学习与数据基础研究所);Robotics Institute Germany(德国机器人研究所)
Differentiable Scaffolding Tree for Molecular Optimization
Amplitude(Amplitude公司);Massachusetts Institute of Technology(麻省理工学院);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Georgia Institute of Technology(佐治亚理工学院)