Graph neural networks (GNNs) are gaining popularity for processing graph-structured data. In real-world scenarios, graph data within the same dataset can vary significantly in scale. This variability leads to depth-sensitivity, where the optimal depth of GNN layers depends on the scale of the graph data. Empirically, fewer layers are sufficient for message passing in smaller graphs, while larger graphs typically require deeper networks to capture long-range dependencies and global features. However, existing methods generally use a fixed number of GNN layers to generate representations for all graphs, overlooking the depth-sensitivity issue in graph structure data. To address this challenge, we propose the depth adaptive mixture of expert (DA-MoE) method, which incorporates two main improvements to GNN backbone: \textbf{1)} DA-MoE employs different GNN layers, each considered an expert with its own parameters. Such a design allows the model to flexibly aggregate information at different scales, effectively addressing the depth-sensitivity issue in graph data. \textbf{2)} DA-MoE utilizes GNN to capture the structural information instead of the linear projections in the gating network. Thus, the gating network enables the model to capture complex patterns and dependencies within the data. By leveraging these improvements, each expert in DA-MoE specifically learns distinct graph patterns at different scales. Furthermore, comprehensive experiments on the TU dataset and open graph benchmark (OGB) have shown that DA-MoE consistently surpasses existing baselines on various tasks, including graph, node, and link-level analyses. The code are available at \url{https://github.com/Celin-Yao/DA-MoE}.
University of Oxford(牛津大学);Mila - Québec AI Institute(米拉-魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);McGill University(麦吉尔大学);Google DeepMind(谷歌DeepMind)
MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Peng Cheng Laboratory(鹏城实验室)
College of Cyber Security, Jinan University(暨南大学网络空间安全学院);College of Mathematics and Computer Science, Shantou University(汕头大学数学与计算机科学学院);School of Information Engineering, Guangdong Eco-Engineering Polytechnic(广东生态工程职业学院信息工程学院);Department of Computer Science, University of Illinois Chicago(芝加哥伊利诺伊大学计算机科学系)
EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Nanjing University of Science and Technology(南京理工大学);AIRIA
MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts
School of Electronic and Computer Engineering, Peking University(北京大学电子与计算机工程学院);Peng Cheng Laboratory(鹏城实验室);Peking University Shenzhen Graduate School(北京大学深圳研究生院);Kunlun 2050 Research(昆仑2050研究院);Skywork AI(智源天工AI公司)
DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
University of Science and Technology of China(中国科学技术大学);Tsinghua University(清华大学);Nanyang Technological University(南洋理工大学);University of California, Berkeley(加州大学伯克利分校);University of Washington(华盛顿大学)
TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
Tennessee Tech University, \ Alamos National Laboratory Cookeville Tennessee USA;Los Alamos National Laboratory Los Alamos New Mexico USA;Tennessee Tech University Cookeville Tennessee USA;Tennessee Tech University, \ Alamos National Laboratory;Los Alamos National Laboratory;Tennessee Tech University
Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
University of Warsaw(华沙大学);Wroclaw University of Science and Technology(弗罗茨瓦夫理工大学);Institute of Mathematics, Polish Academy of Sciences(波兰科学院数学研究所);Institute of Fundamental Technological Research, Polish Academy of Sciences(波兰科学院基础技术研究所)
S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);School of Integrated Circuits, Peking University(北京大学集成电路学院);School of Electronics Engineering and Computer Science, Peking University(北京大学电子工程与计算机科学学院)
ECG-MoE: Mixture-of-Expert Electrocardiogram Foundation Model
Department of Computer Science(计算机科学系);Emory University(埃默里大学);School of Computer Science(计算机科学学院);University of Oklahoma(俄克拉荷马大学);Center of Data Science(数据科学中心)
Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference
National Key Laboratory of Parallel and Distributed Computing, College of Computer Science and Technology, National University of Defense Technology(国防科技大学并行与分布计算国家重点实验室,计算机科学与技术学院)
Booth School of Business University of Chicago(芝加哥大学商学院);Volgenau School of Engineering, George Mason University(乔治·马歇尔大学工程学院);Department of Systems Engineering and Operations Research George Mason University(乔治·马歇尔大学系统工程与运营管理系)
Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Baidu Inc.(百度公司)