We aim to design strategies for sequential decision making that adjust to the difficulty of the learning problem. We study this question both in the setting of prediction with expert advice, and for more general combinatorial decision tasks. We are not satisfied with just guaranteeing minimax regret rates, but we want our algorithms to perform significantly better on easy data. Two popular ways to formalize such adaptivity are second-order regret bounds and quantile bounds. The underlying notions of 'easy data', which may be paraphrased as "the learning problem has small variance" and "multiple decisions are useful", are synergetic. But even though there are sophisticated algorithms that exploit one of the two, no existing algorithm is able to adapt to both.
In this paper we outline a new method for obtaining such adaptive algorithms, based on a potential function that aggregates a range of learning rates (which are essential tuning parameters). By choosing the right prior we construct efficient algorithms and show that they reap both benefits by proving the first bounds that are both second-order and incorporate quantiles.
University of Toronto（多伦多大学）；Spanflug Technologies GmbH（Spanflug技术有限公司）；Boston University（波士顿大学）
Quantile (and, more generally, KL) regret bounds, such as those achieved by NormalHedge (Chaudhuri, Freund, and Hsu 2009) and its variants, relax the goal of competing against the best individual expert to only competing against a majority of experts on adversarial data. More recently, the semi-adversarial paradigm (Bilodeau, Negrea, and Roy 2020) provides an alternative relaxation of adversarial online learning by considering data that may be neither fully adversarial nor stochastic (i.i.d.). We achieve the minimax optimal regret in both paradigms using FTRL with separate, novel, root-logarit…
University of British Columbia（不列颠哥伦比亚大学）；Google（谷歌）
Prediction with experts' advice is one of the most fundamental problems in online learning and captures many of its technical challenges. A recent line of work has looked at online learning through the lens of differential equations and continuous-time analysis. This viewpoint has yielded optimal results for several problems in online learning. In this paper, we employ continuous-time stochastic calculus in order to study the discrete-time experts' problem. We use these tools to design a continuous-time, parameter-free algorithm with improved guarantees for the quantile regret. We then develop…
In order to find Nash-equilibria for two-player zero-sum games where each player plays combinatorial objects like spanning trees, matchings etc, we consider two online learning algorithms: the online mirror descent (OMD) algorithm and the multiplicative weights update (MWU) algorithm. The OMD algorithm requires the computation of a certain Bregman projection, that has closed form solutions for simple convex sets like the Euclidean ball or the simplex. However, for general polyhedra one often needs to exploit the general machinery of convex optimization. We give a novel primal-style algorithm f…
Divide and Learn: Multi-Objective Combinatorial Optimization at Scale
Department of Computer Science & Engineering UC San Diego（计算机科学与工程系，圣地亚哥大学）；Department of Statistics Stanford University（统计学系，斯坦福大学）；Halıcıoğlu Data Science Institute UC San Diego（Halıcıoğlu数据科学研究所，圣地亚哥大学）
IBM Research（IBM研究）；Argonne National Laboratory（阿贡国家实验室）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
We consider the problem of black-box function optimization over the boolean hypercube. Despite the vast literature on black-box function optimization over continuous domains, not much attention has been paid to learning models for optimization over combinatorial domains until recently. However, the computational complexity of the recently devised algorithms are prohibitive even for moderate numbers of variables; drawing one sample using the existing algorithms is more expensive than a function evaluation for many black-box functions of interest. To address this problem, we propose a computatio…
University of Southern California（南加利福尼亚大学）；Amazon Web Services（亚马逊云科技）
Despite impressive state-of-the-art performance on a wide variety of machine learning tasks, deep learning methods can produce over-confident predictions, particularly with limited training data. Therefore, quantifying uncertainty is particularly important in critical applications such as lesion detection and clinical diagnosis, where a realistic assessment of uncertainty is essential in determining surgical margins, disease status and appropriate treatment. In this work, we propose a novel approach that uses quantile regression for quantifying aleatoric uncertainty in both supervised and unsu…
Columbia University（哥伦比亚大学）；Cornell University（康奈尔大学）
Combinatorial optimization algorithms for graph problems are usually designed afresh for each new problem with careful attention by an expert to the problem structure. In this work, we develop a new framework to solve any combinatorial optimization problem over graphs that can be formulated as a single player game defined by states, actions, and rewards, including minimum spanning tree, shortest paths, traveling salesman problem, and vehicle routing problem, without expert knowledge. Our method trains a graph neural network using reinforcement learning on an unlabeled training set of graphs. T…
Korea Advanced Institute of Science and Technology (KAIST)（韩国高等科学技术研究院（KAIST））
Deep reinforcement learning (DRL)-based combinatorial optimization (CO) methods (i.e., DRL-NCO) have shown significant merit over the conventional CO solvers as DRL-NCO is capable of learning CO solvers less relying on problem-specific expert domain knowledge (heuristic method) and supervised labeled data (supervised learning method). This paper presents a novel training scheme, Sym-NCO, which is a regularizer-based training scheme that leverages universal symmetricities in various CO problems and solutions. Leveraging symmetricities such as rotational and reflectional invariance can greatly i…
Mila（米拉研究所）；Polytechnique Montréal（蒙特利尔理工学院）；University of Bologna（博洛尼亚大学）；HEC Montréal（蒙特利尔高等商学院）
Combinatorial optimization problems are typically tackled by the branch-and-bound paradigm. We propose a new graph convolutional neural network model for learning branch-and-bound variable selection policies, which leverages the natural variable-constraint bipartite graph representation of mixed-integer linear programs. We train our model via imitation learning from the strong branching expert rule, and demonstrate on a series of hard problems that our approach produces policies that improve upon state-of-the-art machine-learning methods for branching and generalize to instances significantly …
California Institute of Technology（加州理工学院）；Jet Propulsion Laboratory（喷气推进实验室）
We study the problem of learning a good search policy for combinatorial search spaces. We propose retrospective imitation learning, which, after initial training by an expert, improves itself by learning from \textit{retrospective inspections} of its own roll-outs. That is, when the policy eventually reaches a feasible solution in a combinatorial search tree after making mistakes and backtracks, it retrospectively constructs an improved search trace to the solution by removing backtracks, which is then used to further train the policy. A key feature of our approach is that it can iteratively s…
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）；Zyfra（兹夫特拉）；Criteo AI Lab（Criteo人工智能实验室）
Many traditional algorithms for solving combinatorial optimization problems involve using hand-crafted heuristics that sequentially construct a solution. Such heuristics are designed by domain experts and may often be suboptimal due to the hard nature of the problems. Reinforcement learning (RL) proposes a good alternative to automate the search of these heuristics by training an agent in a supervised or self-supervised manner. In this survey, we explore the recent advancements of applying RL frameworks to hard combinatorial problems. Our survey provides the necessary background for operations…
Many combinatorial optimization problems are often considered intractable to solve exactly or by approximation. An example of such problem is maximum clique which -- under standard assumptions in complexity theory -- cannot be solved in sub-exponential time or be approximated within polynomial factor efficiently. We show that if a polynomial time algorithm can query informative Gaussian priors from an expert $poly(n)$ times, then a class of combinatorial optimization problems can be solved efficiently in expectation up to a multiplicative factor $ε$ where $ε$ is arbitrary constant. While our p…
Rethinking Neural Combinatorial Optimization for Vehicle Routing Problems with Different Constraint Tightness Degrees
School of Automation and Intelligent Manufacturing, Southern University of Science and Technology, Shenzhen, China（自动化与智能制造学院，南方科技大学，深圳，中国）；Guangdong Provincial Key Laboratory of Fully Actuated System Control Theory and Technology, Southern University of Science and Technology, Shenzhen, China（广东省全驱动系统控制理论与技术重点实验室，南方科技大学，深圳，中国）；Department of Industrial Engineering and Innovation Sciences, Eindhoven University of Technology, Eindhoven, The Netherlands（工业工程与创新科学系，埃因霍温理工大学，埃因霍温，荷兰）；School of Computing, National University of Singapore, Singapore（计算学院，新加坡国立大学）
Adversarial self-play in two-player games has delivered impressive results when used with reinforcement learning algorithms that combine deep neural networks and tree search. Algorithms like AlphaZero and Expert Iteration learn tabula-rasa, producing highly informative training data on the fly. However, the self-play training strategy is not directly applicable to single-player games. Recently, several practically important combinatorial optimisation problems, such as the travelling salesman problem and the bin packing problem, have been reformulated as reinforcement learning problems, increas…
Preference Optimization for Combinatorial Optimization Problems
China Mobile（中国移动）；Peking University（北京大学）；Tsinghua University（清华大学）；Jiaying University（嘉应学院）；Sun Yat-Sen University（中山大学）；Central South University（中南大学）
McGill University（麦吉尔大学）；ICREA；Pompeu Fabra University（庞培法布拉大学）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）；Computer and Automation Research Institute of the Hungarian Academy of Sciences（匈牙利科学院计算机与自动化研究所）
We propose a version of the follow-the-perturbed-leader online prediction algorithm in which the cumulative losses are perturbed by independent symmetric random walks. The forecaster is shown to achieve an expected regret of the optimal order O(sqrt(n log N)) where n is the time horizon and N is the number of experts. More importantly, it is shown that the forecaster changes its prediction at most O(sqrt(n log N)) times, in expectation. We also extend the analysis to online combinatorial optimization and show that even in this more general setting, the forecaster rarely switches between expert…
RoME: Domain-Robust Mixture-of-Experts for MILP Solution Prediction across Domains
Laboratory for Big Data and Decision, College of Systems Engineering, National University of Defense Technology（大数据与决策实验室，系统工程学院，国防科技大学）；MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China（脑启发智能感知与认知关键实验室，中国科学技术大学）；College of Computer Science and Technology, National University of Defense Technology（计算机科学与技术学院，国防科技大学）
MIT Sloan School of Management（麻省理工学院斯隆管理学院）；MIT Operations Research Center（麻省理工学院运筹学研究中心）；MIT Center for Statistics（麻省理工学院统计中心）
Decision trees are one of the most useful and popular methods in the machine learning toolbox. In this paper, we consider the problem of learning optimal decision trees, a combinatorial optimization problem that is challenging to solve at scale. A common approach in the literature is to use greedy heuristics, which may not be optimal. Recently there has been significant interest in learning optimal decision trees using various approaches (e.g., based on integer programming, dynamic programming) -- to achieve computational scalability, most of these approaches focus on classification tasks with…
RL4CO: an Extensive Reinforcement Learning for Combinatorial Optimization Benchmark
Leuphana University（吕贝克大学）；MIT（麻省理工学院）；Southeast University（东南大学）；Peking University（北京大学）；Bielefeld University（比勒菲尔德大学）；Nanyang Technological University（南洋理工大学）；Soochow University（苏州大学）；City University of Hong Kong（香港城市大学）；University of Brescia（布雷西亚大学）；ORTEC（ORTEC公司）；Singapore Management University（新加坡管理学院）；Brandenburg University of Technology（勃兰登堡技术大学）
Refining Hybrid Genetic Search for CVRP via Reinforcement Learning-Finetuned LLM
School of Teacher Education, Nanjing University of Information Science and Technology, China（南京信息工程大学教师教育学院）；Nanyang Technological University, Singapore（新加坡南洋理工大学）；Singapore Management University, Singapore（新加坡管理大学）
