Solving Mixed Integer Programs Using Neural Networks
DeepMind；Google Research（谷歌研究院）
Vinod Nair, Sergey Bartunov, Felix Gimeno, Ingrid von Glehn, Pawel Lichocki, Ivan Lobov, Brendan O'Donoghue, Nicolas Sonnerat, Christian Tjandraatmadja, Pengmin… 展开作者
Vinod Nair, Sergey Bartunov, Felix Gimeno, Ingrid von Glehn, Pawel Lichocki, Ivan Lobov, Brendan O'Donoghue, Nicolas Sonnerat, Christian Tjandraatmadja, Pengming Wang, Ravichandra Addanki, Tharindi Hapuarachchi, Thomas Keck, James Keeling, Pushmeet Kohli, Ira Ktena, Yujia Li, Oriol Vinyals, Yori Zwols
Mixed Integer Programming (MIP) solvers rely on an array of sophisticated heuristics developed with decades of research to solve large-scale MIP instances encountered in practice. Machine learning offers to automatically construct better heuristics from data by exploiting shared structure among instances in the data. This paper applies learning to the two key sub-tasks of a MIP solver, generating a high-quality joint variable assignment, and bounding the gap in objective value between that assignment and an optimal one. Our approach constructs two corresponding neural network-based components, Neural Diving and Neural Branching, to use in a base MIP solver such as SCIP. Neural Diving learns a deep neural network to generate multiple partial assignments for its integer variables, and the resulting smaller MIPs for un-assigned variables are solved with SCIP to construct high quality joint assignments. Neural Branching learns a deep neural network to make variable selection decisions in branch-and-bound to bound the objective value gap with a small tree. This is done by imitating a new variant of Full Strong Branching we propose that scales to large instances using GPUs. We evaluate our approach on six diverse real-world datasets, including two Google production datasets and MIPLIB, by training separate neural networks on each. Most instances in all the datasets combined have $10^3-10^6$ variables and constraints after presolve, which is significantly larger than previous learning approaches. Comparing solvers with respect to primal-dual gap averaged over a held-out set of instances, the learning-augmented SCIP is 2x to 10x better on all datasets except one on which it is $10^5$x better, at large time limits. To the best of our knowledge, ours is the first learning approach to demonstrate such large improvements over SCIP on both large-scale real-world application datasets and MIPLIB.
Cutting planes are a key ingredient to successfully solve mixed-integer linear programs. For specific problems, their strength is often theoretically assessed by showing that they are facet-defining for the corresponding mixed-integer hull. In this paper we experimentally investigate the dimensions of faces induced by general-purpose cutting planes generated by a state-of-the-art solver. Therefore, we relate the dimension of each cutting plane to its impact in a branch-and-bound algorithm.
Multi-objective integer or mixed-integer programming problems typically have disconnected feasible domains, making the task of constructing an approximation of the Pareto front challenging. The present paper shows that certain algorithms which were originally devised for continuous problems can be successfully adapted to approximate the Pareto front for integer, and mixed-integer, multi-objective optimization problems. Relationships amongst various scalarization techniques are established to motivate the choice of a particular scalarization in these algorithms. The proposed algorithms are test…
We study the reformulation of integer linear programs by means of a mixed integer linear program with fewer integer variables. Such reformulations can be solved efficiently with mixed integer linear programming techniques. We exhibit examples that demonstrate how integer programs can be reformulated using far fewer integer variables. To this end, we introduce a generalization of total unimodularity called the \emph{affine TU-dimension} of a matrix and study related theory and algorithms for determining the affine TU-dimension of a matrix. We also present bounds on the number of integer variabl…
In this paper, we present convex hull formulations for a mixed-integer, multilinear term/function (MIMF) that features products of multiple continuous and binary variables. We develop two equivalent convex relaxations of an MIMF and study their polyhedral properties in their corresponding higher-dimensional spaces. We numerically observe that the proposed formulations consistently perform better than state-of-the-art relaxation approaches.
Mixed-integer mathematical programs are among the most commonly used models for a wide set of problems in Operations Research and related fields. However, there is still very little known about what can be expressed by small mixed-integer programs. In particular, prior to this work, it was open whether some classical problems, like the minimum odd-cut problem, can be expressed by a compact mixed-integer program with few (even constantly many) integer variables. This is in stark contrast to linear formulations, where recent breakthroughs in the field of extended formulations have shown that man…
We consider the constrained assortment optimization problem under the mixed multinomial logit model. Even moderately sized instances of this problem are challenging to solve directly using standard mixed-integer linear optimization formulations. This has motivated recent research exploring customized optimization strategies and approximation techniques. In contrast, we develop a novel conic quadratic mixed-integer formulation. This new formulation, together with McCormick inequalities exploiting the capacity constraints, enables the solution of large instances using commercial optimization sof…
In this paper, we give an algorithm that finds an epsilon-approximate solution to a mixed integer quadratic programming (MIQP) problem. The algorithm runs in polynomial time if the rank of the quadratic function and the number of integer variables are fixed. The running time of the algorithm is expected unless P=NP. In order to design this algorithm we introduce the novel concepts of spherical form MIQP and of aligned vectors, and we provide a number of results of independent interest. In particular, we give a strongly polynomial algorithm to find a symmetric decomposition of a matrix, and sho…
We study the representability of sets that admit extended formulations using mixed-integer bilevel programs. We show that feasible regions modeled by continuous bilevel constraints (with no integer variables), complementarity constraints, and polyhedral reverse convex constraints are all finite unions of polyhedra. Conversely, any finite union of polyhedra can be represented using any one of these three paradigms. We then prove that the feasible region of bilevel problems with integer constraints exclusively in the upper level is a finite union of sets representable by mixed-integer programs a…
We propose a dual dynamic integer programming (DDIP) framework for solving multi-scale mixed-integer model predictive control (MPC) problems. Such problems arise in applications that involve long horizons and/or fine temporal discretizations as well as mixed-integer states and controls (e.g., scheduling logic and discrete actuators). The approach uses a nested cutting-plane scheme that performs forward and backward sweeps along the time horizon to adaptively approximate cost-to-go functions. The DDIP scheme proposed can handle general MPC formulations with mixed-integer controls and states and…
This paper presents a mixed-integer quadratic programming formulation of an existing data-driven approach to computational elasticity. This formulation is suitable for application of a standard mixed-integer programming solver, which finds a global optimal solution. Therefore, the results obtained by the presented method can be used as benchmark instances for any other algorithm. Preliminary numerical experiments are performed to compare quality of solutions obtained by the proposed method and a heuristic used in the data-driven computational mechanics.
We consider mixed-integer optimal control problems with combinatorial constraints that couple over time such as minimum dwell times. We analyze a lifting and decomposition approach into a mixed-integer optimal control problem without combinatorial constraints and a mixed-integer problem for the combinatorial constraints in the control space. Both problems can be solved very efficiently with existing methods such as outer convexification with sum-up-rounding strategies and mixed-integer linear programming techniques. The coupling is handled using a penalty-approach. We provide an exactness resu…
We consider the question of which nonconvex sets can be represented exactly as the feasible sets of mixed-integer convex optimization problems. We state the first complete characterization for the case when the number of possible integer assignments is finite. We develop a characterization for the more general case of unbounded integer variables together with a simple necessary condition for representability which we use to prove the first known negative results. Finally, we study representability of subsets of the natural numbers, developing insight towards a more complete understanding of wh…
We consider the problem of computing a mixed-strategy generalized Nash equilibrium (MS-GNE) for a class of games where each agent has both continuous and integer decision variables. Specifically, we propose a novel Bregman forward-reflected-backward splitting and design distributed algorithms that exploit the problem structure. Technically, we prove convergence to a variational MS-GNE under mere monotonicity and Lipschitz continuity assumptions, which are typical of continuous GNE problems. Finally, we show the performance of our algorithms via numerical experiments.
The technique of semidefinite programming (SDP) relaxation can be used to obtain a nontrivial bound on the optimal value of a nonconvex quadratically constrained quadratic program (QCQP). We explore concave quadratic inequalities that hold for any vector in the integer lattice ${\bf Z}^n$, and show that adding these inequalities to a mixed-integer nonconvex QCQP can improve the SDP-based bound on the optimal value. This scheme is tested using several numerical problem instances of the max-cut problem and the integer least squares problem.
We study a canonical duality method to solve a mixed-integer nonconvex fourth-order polynomial minimization problem with fixed cost terms. This constrained nonconvex problem can be transformed into a continuous concave maximization dual problem without duality gap. The global optimality conditions are proposed and the existence and uniqueness criteria are discussed. Application to a decoupled mixed-integer problem is illustrated and analytic solution for a global minimum is obtained under some suitable conditions. Several examples are given to show the method is effective.
Quantum computing is emerging as a new computing resource that could be superior to conventional computing for certain classes of optimization problems. However, in principle, most existing approaches to quantum optimization are intended to solve unconstrained binary programming problems, while mixed-integer linear programming is of most interest in practice. We attempt to bridge the gap between the capability of quantum computing and real-world applications by developing a new approach for mixed-integer programming. The approach applies Benders decomposition to decompose the mixed-integer pro…
In this work, we present a modeling framework for minimax mixed 0-1 fuzzy linear problems. It is based on extending the usual rewriting of crisp minimax problems via auxiliary variables to model the maximum of a finite set of fuzzy linear functions. We establish that the considered problem can be equivalently formulated as a multiple objective mixed integer programming problem. The framework is applied to a fully fuzzy version of the capacitated center facility location problem.
Data-driven Mixed Integer Optimization through Probabilistic Multi-variable Branching
Shanghai University of Finance and Economics（上海财经大学）；Stanford University（斯坦福大学）；Antai College of Economics and Management, Shanghai Jiao Tong University（上海交通大学安泰经济与管理学院）
An important problem in optimization is the construction of mixed-integer programming (MIP) formulations of disjunctive constraints that are both strong and small. Motivated by lower bounds on the number of integer variables that are required by traditional MIP formulations, we present a more general mixed-integer branching formulation framework. Our approach maintains favorable algorithmic properties of traditional MIP formulations: in particular, amenability to branch-and-bound and branch-and-cut algorithms. Our main technical result gives an explicit linear inequality description for both t…
Motivated by recent advances in solution methods for mixed-integer convex optimization (MICP), we study the fundamental and open question of which sets can be represented exactly as feasible regions of MICP problems. We establish several results in this direction, including the first complete characterization for the mixed-binary case and a simple necessary condition for the general case. We use the latter to derive the first non-representability results for various non-convex sets such as the set of rank-1 matrices and the set of prime numbers. Finally, in correspondence with the seminal work…
Mix-CALADIN: A Distributed Algorithm for Consensus Mixed-Integer Optimization
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；KTH Royal Institute of Technology（瑞典皇家理工学院）；The Hong Kong University of Science and Technology（香港科技大学）
This note discusses properties of parametric discrete-time Mixed-Integer Optimal Control Problems (MIOCPs) as they often arise in mixed-integer NMPC. We argue that in want for a handle on similarity properties of parametric MIOCPs the classical turnpike notion from optimal control is helpful. We provide sufficient turnpike conditions based on a dissipativity notion of MIOCPs, and we show that the turnpike property allows specific and accurate guesses for the integer-valued controls. Moreover, we show how the turnpike property can be used to derive efficient node-weighted branch-and-bound schem…
Artificial Neural Networks (ANNs) are prevalent machine learning models that are applied across various real-world classification tasks. However, training ANNs is time-consuming and the resulting models take a lot of memory to deploy. In order to train more parsimonious ANNs, we propose a novel mixed-integer programming (MIP) formulation for training fully-connected ANNs. Our formulations can account for both binary and rectified linear unit (ReLU) activations, and for the use of a log-likelihood loss. We present numerical experiments comparing our MIP-based methods against existing approaches…
Two-stage stochastic mixed-integer programming (SMIP) problems with general integer variables in the second-stage are generally difficult to solve. This paper develops the theory of integer set reduction for characterizing the subset of the convex hull of feasible integer points of the second-stage subproblem which can be used for solving the SMIP. The basic idea is to consider a small enough subset of feasible integer points that is necessary for generating a valid inequality for the integer subproblem. An algorithm for obtaining such a subset based on the solution of the subproblem LP-relaxa…
We prove that any mixed-integer linear extended formulation for the matching polytope of the complete graph on $n$ vertices, with a polynomial number of constraints, requires $Ω(\sqrt{\sfrac{n}{\log n}})$ many integer variables. By known reductions, this result extends to the traveling salesman polytope. This lower bound has various implications regarding the existence of small mixed-integer mathematical formulations of common problems in operations research. In particular, it shows that for many classic vehicle routing problems and problems involving matchings, any compact mixed-integer linea…
A classic result of Cook et al. (1986) bounds the distances between optimal solutions of mixed-integer linear programs and optimal solutions of the corresponding linear relaxations. Their bound is given in terms of the number of variables and a parameter $ Δ$, which quantifies sub-determinants of the underlying linear inequalities. We show that this distance can be bounded in terms of $ Δ$ and the number of integer variables rather than the total number of variables. To this end, we make use of a result by Olson (1969) in additive combinatorics and demonstrate how it implies feasibility of cer…
We give an explicit geometric way to build mixed-integer programming (MIP) formulations for unions of polyhedra. The construction is simply described in terms of spanning hyperplanes in an r-dimensional linear space. The resulting MIP formulation is ideal, and uses exactly r integer variables and 2 x (# of spanning hyperplanes) general inequality constraints. We use this result to derive novel logarithmic-sized ideal MIP formulations for discontinuous piecewise linear functions and structures appearing in robotics and power systems problems.
Machine Learning Algorithms for Improving Exact Classical Solvers in Mixed Integer Continuous Optimization
Fakultät für Mathematik, Universität Wien（维也纳大学数学系）；Department of Computer Science, Czech Technical University（捷克技术大学计算机科学系）；CEO Olicheza Limited（Olicheza有限公司）
Argonne National Laboratory（阿贡国家实验室）；University of Minnesota Twin Cities（明尼苏达大学双城分校）
In this paper we consider the problem of learning a regression function without assuming its functional form. This problem is referred to as symbolic regression. An expression tree is typically used to represent a solution function, which is determined by assigning operators and operands to the nodes. The symbolic regression problem can be formulated as a nonconvex mixed-integer nonlinear program (MINLP), where binary variables are used to assign operators and nonlinear expressions are used to propagate data values through nonlinear operators such as square, square root, and exponential. We ex…
In this paper, we show that the subadditive dual of a feasible conic mixed-integer program (MIP) is a strong dual whenever it is feasible. Moreover, we show that this dual feasibility condition is equivalent to feasibility of the conic dual of the continuous relaxation of the conic MIP. In addition, we prove that all known conditions and other 'natural' conditions for strong duality, such as strict mixed-integer feasibility, boundedness of the feasible set or essentially strict feasibility imply that the subadditive dual is feasible. As an intermediate result, we extend the so-called 'finitene…
University of Siegen（锡根大学）；African Institute for Mathematical Sciences（非洲数学科学研究所）
Compared to classical deep neural networks its binarized versions can be useful for applications on resource-limited devices due to their reduction in memory consumption and computational demands. In this work we study deep neural networks with binary activation functions and continuous or integer weights (BDNN). We show that the BDNN can be reformulated as a mixed-integer linear program with bounded weight space which can be solved to global optimality by classical mixed-integer programming solvers. Additionally, a local search heuristic is presented to calculate locally optimal networks. Fur…
This letter presents a fast distributed algorithm for aggregating a large number of households with mixed-integer variables and intricate couplings between devices. The proposed fast distributed gradient algorithm is applied to the double smoothed dual function of the adopted DR model. The results also show that, with minimal parameter adjustments, the convergence of the dual objective exhibits the same behavior irrespective of the system size.
Decision trees usefully represent sparse, high dimensional and noisy data. Having learned a function from this data, we may want to thereafter integrate the function into a larger decision-making problem, e.g., for picking the best chemical process catalyst. We study a large-scale, industrially-relevant mixed-integer nonlinear nonconvex optimization problem involving both gradient-boosted trees and penalty functions mitigating risk. This mixed-integer optimization problem with convex penalty terms broadly applies to optimizing pre-trained regression tree models. Decision makers may wish to opt…
Endogenous, i.e. decision-dependent, uncertainty has received increased interest in the stochastic programming community. In the robust optimization context, however, it has rarely been considered. This work addresses multistage robust mixed-integer optimization with decision-dependent uncertainty sets. The proposed framework allows us to consider both continuous and integer recourse, including recourse decisions that affect the uncertainty set. We derive a tractable reformulation of the problem by leveraging recent advances in the construction of nonlinear decision rules, and introduce discon…
Solution Seeker AS；Norwegian University of Science and Technology（挪威科技大学）
We consider the embedding of piecewise-linear deep neural networks (ReLU networks) as surrogate models in mixed-integer linear programming (MILP) problems. A MILP formulation of ReLU networks has recently been applied by many authors to probe for various model properties subject to input bounds. The formulation is obtained by programming each ReLU operator with a binary variable and applying the big-M method. The efficiency of the formulation hinges on the tightness of the bounds defined by the big-M values. When ReLU networks are embedded in a larger optimization problem, the presence of outp…
We introduce a unified framework for the study of multilevel mixed integer linear optimization problems and multistage stochastic mixed integer linear optimization problems with recourse. The framework highlights the common mathematical structure of the two problems and allows for the development of a common algorithmic framework. Focusing on the two-stage case, we investigate, in particular, the nature of the value function of the second-stage problem, highlighting its connection to dual functions and the theory of duality for mixed integer linear optimization problems, and summarize differen…
