Mixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained expert segmentation and shared experts. The former increases expressiveness by enabling more expert combinations at a fixed activation ratio, and the latter introduces experts that are active for every token. We identify an exploration dilemma between these designs: shared experts can make routing less exploratory by reducing the variability of token representations. This conflicts with the greater need for exploration induced by fine-grained expert segmentation due to the combinatorially enlarged routing space. Consequently, the additional expressiveness offered by fine-grained segmentation may not be fully realized in practice. MirrorMoE is proposed to reconcile this exploration dilemma by shrinking the exploration space through token-adaptive suppression of unnecessary experts. MirrorMoE consistently improves performance across model scales.
GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanism
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Institute for Advanced Algorithms Research, Shanghai（上海先进算法研究所）；The University of Manchester（曼彻斯特大学）；The University of Pittsburgh（匹兹堡大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Sydney（悉尼大学）
Rethinking Constraint Awareness for Efficient State Embedding of Neural Routing Solver
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；School of Automation and Intelligent Manufacturing, Southern University of Science and Technology（南方科技大学自动化与智能制造学院）；Guangdong Provincial Key Laboratory of Fully Actuated System Control Theory and Technology, Southern University of Science and Technology（广东省全驱动系统控制理论与技术重点实验室，南方科技大学）；Pengcheng Laboratory（鹏城实验室）
We are at the cusp of a transition from "learning from data" to "learning what data to learn from" as a central focus of artificial intelligence (AI) research. While the first-order learning problem is not completely solved, large models under unified architectures, such as transformers, have shifted the learning bottleneck from how to effectively train our models to how to effectively acquire and use task-relevant data. This problem, which we frame as exploration, is a universal aspect of learning in open-ended domains, such as the real world. Although the study of exploration in AI is largel…
Rethinking Light Decoder-based Solvers for Vehicle Routing Problems
School of Computing and Information Systems, Singapore Management University（新加坡管理大学计算与信息系统学院）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）
From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routing
School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；National Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；Hong Kong University of Science and Technology（香港科学与技术大学）；Frontier Robotics（前沿机器人）
MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation
Shanghai Jiao Tong University（上海交通大学）；GAIR；Alibaba Group（阿里巴巴集团）；University of Science and Technology of China（中国科学技术大学）；Shanghai Innovation Institute（上海创新研究院）
Rethinking Neural Combinatorial Optimization for Vehicle Routing Problems with Different Constraint Tightness Degrees
School of Automation and Intelligent Manufacturing, Southern University of Science and Technology, Shenzhen, China（自动化与智能制造学院，南方科技大学，深圳，中国）；Guangdong Provincial Key Laboratory of Fully Actuated System Control Theory and Technology, Southern University of Science and Technology, Shenzhen, China（广东省全驱动系统控制理论与技术重点实验室，南方科技大学，深圳，中国）；Department of Industrial Engineering and Innovation Sciences, Eindhoven University of Technology, Eindhoven, The Netherlands（工业工程与创新科学系，埃因霍温理工大学，埃因霍温，荷兰）；School of Computing, National University of Singapore, Singapore（计算学院，新加坡国立大学）
Current state-of-the-art large-scale conversational AI or intelligent digital assistant systems in industry comprises a set of components such as Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU). For some of these systems that leverage a shared NLU ontology (e.g., a centralized intent/slot schema), there exists a separate skill routing component to correctly route a request to an appropriate skill, which is either a first-party or third-party application that actually executes on a user request. The skill routing component is needed as there are thousands of skills t…
A key challenge in solving a combinatorial optimization problem is how to guide the agent (i.e., solver) to efficiently explore the enormous search space. Conventional approaches often rely on enumeration (e.g., exhaustive, random, or tabu search) or have to restrict the exploration to rather limited regions (e.g., a single path as in iterative algorithms). In this paper, we show it is possible to use machine learning to speedup the exploration. In particular, a value network is trained to evaluate solution candidates, which provides a useful structure (i.e., an approximate value surface) over…
Rethinking LLM Ensembling from the Perspective of Mixture Models
Key Laboratory of New Generation Artificial Intelligence Technology（新一代人工智能技术关键实验室）；Its Interdisciplinary Applications (Southeast University), Ministry of Education（交叉应用（东南大学），教育部）；Southeast University（东南大学）；Centre for Frontier AI Research (CFAR), Agency for Science, Technology（前沿人工智能研究（CFAR），科技研究局）；Research (A STAR), Singapore（研究（A STAR），新加坡）；Institute of High Performance Computing (IHPC), Agency for Science, Technology（高性能计算（IHPC），科技研究局）
Model-free deep-reinforcement-based learning algorithms have been applied to a range of COPs~\cite{bello2016neural}~\cite{kool2018attention}~\cite{nazari2018reinforcement}. However, these approaches suffer from two key challenges when applied to combinatorial problems: insufficient exploration and the requirement of many training examples of the search space to achieve reasonable performance. Combinatorial optimisation can be complex, characterised by search spaces with many optimas and large spaces to search and learn. Therefore, a new method is needed to find good solutions that are more eff…
Physical design process commonly consumes hours to days for large designs, and routing is known as the most critical step. Demands for accurate routing quality prediction raise to a new level to accelerate hardware innovation with advanced technology nodes. This work presents an approach that forecasts the density of all routing channels over the entire floorplan, with features collected up to placement, using conditional GANs. Specifically, forecasting the routing congestion is constructed as an image translation (colorization) problem. The proposed approach is applied to a) placement explora…
For its advantage in GPU acceleration and less dependency on human experts, machine learning has been an emerging tool for solving the placement and routing problems, as two critical steps in modern chip design flow. Being still in its early stage, there are fundamental issues: scalability, reward design, and end-to-end learning paradigm etc. To achieve end-to-end placement learning, we first propose a joint learning method termed by DeepPlace for the placement of macros and standard cells, by the integration of reinforcement learning with a gradient based optimization scheme. To further bridg…
Learning Decentralized Routing Policies via Graph Attention-based Multi-Agent Reinforcement Learning in Lunar Delay-Tolerant Networks
Telecommunications Research Institute, University of Málaga（马德里大学电信研究所）；Jet Propulsion Laboratory, California Institute of Technology（加州理工学院喷气推进实验室）；Department of Mechanical Engineering, Johns Hopkins University（约翰霍普金斯大学机械工程系）
Detailed routing is one of the most critical steps in analog circuit design. Complete routing has become increasingly more challenging in advanced node analog circuits, making advances in efficient automatic routers ever more necessary. In this work, we propose a machine learning driven method for solving the track-assignment detailed routing problem for advanced node analog circuits. Our approach adopts an attention-based reinforcement learning (RL) policy model. Our main insight and advancement over this RL model is the use of supervision as a way to leverage solutions generated by a convent…
Johns Hopkins University（约翰斯·霍普金斯大学）；Stanford University（斯坦福大学）；UC Santa Cruz（加州大学圣克鲁兹分校）；Carnegie Mellon University（卡内基梅隆大学）
Federated learning is an emerging research paradigm enabling collaborative training of machine learning models among different organizations while keeping data private at each institution. Despite recent progress, there remain fundamental challenges such as the lack of convergence and the potential for catastrophic forgetting across real-world heterogeneous devices. In this paper, we demonstrate that self-attention-based architectures (e.g., Transformers) are more robust to distribution shifts and hence improve federated learning over heterogeneous data. Concretely, we conduct the first rigoro…
Hong Kong University of Science and Technology（香港科技大学）；Indian Institute of Technology Goa（印度理工学院果阿分校）
High-level synthesis (HLS) shortens the development time of hardware designs and enables faster design space exploration at a higher abstraction level. Optimization of complex applications in HLS is challenging due to the effects of implementation issues such as routing congestion. Routing congestion estimation is absent or inaccurate in existing HLS design methods and tools. Early and accurate congestion estimation is of great benefit to guide the optimization in HLS and improve the efficiency of implementation. However, routability, a serious concern in FPGA designs, has been difficult to ev…
