The recently presented idea to learn heuristics for combinatorial optimization problems is promising as it can save costly development. However, to push this idea towards practical implementation, we need better models and better ways of training. We contribute in both directions: we propose a model based on attention layers with benefits over the Pointer Network and we show how to train this model using REINFORCE with a simple baseline based on a deterministic greedy rollout, which we find is more efficient than using a value function. We significantly improve over recent learned heuristics for the Travelling Salesman Problem (TSP), getting close to optimal results for problems up to 100 nodes. With the same hyperparameters, we learn strong heuristics for two variants of the Vehicle Routing Problem (VRP), the Orienteering Problem (OP) and (a stochastic variant of) the Prize Collecting TSP (PCTSP), outperforming a wide range of baselines and getting results close to highly optimized and specialized algorithms.
Efficient Content-Based Sparse Attention with Routing Transformers
Google Research(谷歌研究院)
自注意最近已被广泛的序列建模问题。尽管它的有效性,自我注意遭受二次计算和存储器的要求相对于序列长度。降低这种复杂性的成功方法集中在关注本地滑动窗口或独立于内容的一小部分位置。我们的工作提出了学习动态稀疏注意模式,避免分配计算和内存参加内容无关的查询的兴趣。这项工作建立在两条线的研究:它结合了先前的工作基于内容的稀疏注意力的建模灵活性,从本地的,时间稀疏注意力的方法的效率收益。我们的模型,路由Transformer,赋予自我注意力的稀疏路由模块的基础上在线k-均值,同时降低整体的复杂性注意$O\left(n^{1.5}d\right)$从$O\left(n^2d\right)$的序列长度$n$和隐藏的维度$d$。我们表明,我们的模型在Wikitext-103上的语言建模(15.8 vs 18.3困惑度)以及ImageNet-64上的图像生成(3.43 vs 3.44 bits/dim)方面优于可比较的稀疏注意模型,同时使用较少的自我注意层。此外,我们在新发布的PG-19数据集上设置了一个新的最新技术水平,使用在长度为8192的序列上训练的22层Routing Transformer模型获得了33.2的测试困惑度。
FAME: Adaptive Functional Attention with Expert Routing for Function-on-Function Regression
Department of Industrial Engineering(工业工程系);Tsinghua University(清华大学);Department of Industrial and Systems Engineering(工业与系统工程系);University of Iowa(爱荷华大学)
MVMoE: Multi-Task Vehicle Routing Solver with Mixture-of-Experts
College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);School of Computing and Information Systems, Singapore Management University(新加坡管理大学计算与信息系统学院);Institute of Marine Science and Technology, Shandong University(山东大学海洋科学与技术研究院);Singapore Institute of Manufacturing Technology (SIMTech), Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局下属新加坡制造技术研究院)
Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers
Fudan University, Shanghai, China(复旦大学);University of Bath, Bath, United Kingdom(巴斯大学);Oxford Suzhou Centre for Advanced Research, Suzhou, China(牛津苏滁研究中心);Shanghai Innovation Institute, Shanghai, China(上海创新研究院);Department of Computer Science, University of Colorado Boulder, Colorado, USA(计算机科学系,科罗拉多大学博尔德分校);Department of Electrical Engineering and Computer Science, University of Michigan(电气工程与计算机科学系,密歇根大学)
Learning Decentralized Routing Policies via Graph Attention-based Multi-Agent Reinforcement Learning in Lunar Delay-Tolerant Networks
Telecommunications Research Institute, University of Málaga(马德里大学电信研究所);Jet Propulsion Laboratory, California Institute of Technology(加州理工学院喷气推进实验室);Department of Mechanical Engineering, Johns Hopkins University(约翰霍普金斯大学机械工程系)