We study deep neural networks with binary activation functions (BDNN), i.e. the activation function only has two states. We show that the BDNN can be reformulated as a mixed-integer linear program which can be solved to global optimality by classical integer programming solvers. Additionally, a heuristic solution algorithm is presented and we study the model under data uncertainty, applying a two-stage robust optimization approach. We implemented our methods on random and real datasets and show that the heuristic version of the BDNN outperforms classical deep neural networks on the Breast Cancer Wisconsin dataset while performing worse on random data.
An example of an activation function $σ$ is given such that networks with activations $\{σ, \lfloor\cdot\rfloor\}$, integer weights and a fixed architecture depending on $d$ approximate continuous functions on $[0,1]^d$. The range of integer weights required for $\varepsilon$-approximation of Hölder continuous functions is derived, which leads to a convergence rate of order $n^{\frac{-2β}{2β+d}}\log_2n$ for neural network regression estimation of unknown $β$-Hölder continuous function with given $n$ samples.
The Hong Kong University of Science and Technology（香港科技大学）
Standard deep learning algorithms are implemented using floating-point real numbers. This presents an obstacle for implementing them on low-end devices which may not have dedicated floating-point units (FPUs). As a result, researchers in tinyML have considered machine learning algorithms that can train and run a deep neural network (DNN) on a low-end device using integer operations only. In this paper we propose PocketNN, a light and self-contained proof-of-concept framework in pure C++ for the training and inference of DNNs using only integers. Unlike other approaches, PocketNN directly opera…
Farkas' Lemma is a foundational result in linear programming, with implications in duality, optimality conditions, and stochastic and bilevel programming. Its generalizations are known as theorems of the alternative. There exist theorems of the alternative for integer programming and conic programming. We present theorems of the alternative for conic integer programming. We provide a nested procedure to construct a function that characterizes feasibility over right-hand sides and can determine which statement in a theorem of the alternative holds.
TU Darmstadt（达姆施塔特工业大学）；TU Berlin（柏林工业大学）；Zuse Institute Berlin（柏林祖斯研究所）
Recently non-convex optimization approaches for solving machine learning problems have gained significant attention. In this paper we explore non-convex boosting in classification by means of integer programming and demonstrate real-world practicability of the approach while circumventing shortcomings of convex boosting approaches. We report results that are comparable to or better than the current state-of-the-art.
In this paper, some deficiencies of a method proposed by Burdet and Johnson in 1977 for solving integer programming problems are discussed. Examples where the algorithm fails to solve the IP and ways to fix these errors are given.
Binary Integer Programming (BIP) problems are of interest due in part to the difficulty they pose and because of their various applications, including those in graph theory, combinatorial optimization and network optimization. In this note, we explicitly state the Lovasz-Schrijver Semidefinite Programming (SDP) relaxation (in primal-standard form) for a BIP problem, a relaxation that yields a tighter upper-bound than the canonical Linear Programming relaxation.
The objective scaling ensemble approach is a novel two-phase heuristic for integer linear programming problems shown to be effective on a wide variety of integer linear programming problems. The technique identifies and aggregates multiple partial solutions to modify the problem formulation and significantly reduce the search space. An empirical analysis on publicly available benchmark problems demonstrate the efficacy of our approach by outperforming standard solution strategies implemented in modern optimization software.
Multi-objective integer or mixed-integer programming problems typically have disconnected feasible domains, making the task of constructing an approximation of the Pareto front challenging. The present paper shows that certain algorithms which were originally devised for continuous problems can be successfully adapted to approximate the Pareto front for integer, and mixed-integer, multi-objective optimization problems. Relationships amongst various scalarization techniques are established to motivate the choice of a particular scalarization in these algorithms. The proposed algorithms are test…
Artificial Neural Networks (ANNs) are prevalent machine learning models that are applied across various real-world classification tasks. However, training ANNs is time-consuming and the resulting models take a lot of memory to deploy. In order to train more parsimonious ANNs, we propose a novel mixed-integer programming (MIP) formulation for training fully-connected ANNs. Our formulations can account for both binary and rectified linear unit (ReLU) activations, and for the use of a log-likelihood loss. We present numerical experiments comparing our MIP-based methods against existing approaches…
Symmetry in integer programs (IPs) can be exploited in order to reduce solving times. Usually only symmetries of the original IP are handled, but new symmetries may arise at some nodes of the branch-and-bound tree. While symmetry-handling inequalities (SHIs) can easily be used to handle original symmetries, handling sub-symmetries arising later on is more intricate. To handle sub-symmetries, it has recently been proposed to add SHIs that are activated by auxiliary variables. This, however, may increase the size of the IP substantially as all sub-symmetries need to be modeled explicitly. As an …
We provide a complexity classification of four variants of robust integer programming when the underlying Graver basis is given. We discuss applications to robust multicommodity flows and multidimensional transportation, and describe an effective parametrization of robust integer programming.
Quantum computing is emerging as a new computing resource that could be superior to conventional computing for certain classes of optimization problems. However, in principle, most existing approaches to quantum optimization are intended to solve unconstrained binary programming problems, while mixed-integer linear programming is of most interest in practice. We attempt to bridge the gap between the capability of quantum computing and real-world applications by developing a new approach for mixed-integer programming. The approach applies Benders decomposition to decompose the mixed-integer pro…
In this paper we develop an algorithm to optimise a nonlinear utility function of multiple objectives over the integer efficient set. Our approach is based on identifying and updating bounds on the individual objectives as well as the optimal utility value. This is done using already known solutions, linear programming relaxations, utility function inversion, and integer programming. We develop a general optimisation algorithm for use with k objectives, and we illustrate our approach using a tri-objective integer programming problem.
We propose a new exact approach for solving integer linear programming (ILP) problems which we will call projective splitting algorithms (PSAs). Unlike classical methods for solving ILP problems, PSAs conduct the search for the optimal solution by generating candidate solutions tailored to specific values of the objective function. As a consequence of this strategy, the number of variables in the original ILP problem is systematically reduced without adding any additional constraint to the initial formulation. This is the first of a two-part series on PSAs. In this paper we focus on the resolu…
This paper presents a mixed-integer quadratic programming formulation of an existing data-driven approach to computational elasticity. This formulation is suitable for application of a standard mixed-integer programming solver, which finds a global optimal solution. Therefore, the results obtained by the presented method can be used as benchmark instances for any other algorithm. Preliminary numerical experiments are performed to compare quality of solutions obtained by the proposed method and a heuristic used in the data-driven computational mechanics.
In this paper we give a new aggregation framework for linear Diophantine equations. In particular, we prove that an aggregated system of minimum size can be built in polynomial time. We also derive an analytic formula that gives the number of solutions of the system when it is possible to aggregate the system into one equation.
Inspired by a paper of R. W. Rosenthal, we investigate generalized Nash-equilibria of integer programming games. We show that generalized Nash-equilibria always exist and are related to an optimal solution of a so-called N-fold integer program. This link allows us to establish some polynomial time complexity results about solving this optimization problem and its inverse counter-part.
The infinite models in integer programming can be described as the convex hull of some points or as the intersection of halfspaces derived from valid functions. In this paper we study the relationships between these two descriptions. Our results have implications for corner polyhedra. One consequence is that nonnegative, continuous valid functions suffice to describe corner polyhedra (with or without rational data).
We overview our recently introduced theory of n-fold integer programming which enables the polynomial time solution of fundamental linear and nonlinear integer programming problems in variable dimension. We demonstrate its power by obtaining the first polynomial time algorithms in several application areas including multicommodity flows and privacy in statistical databases.
The extraction and understanding of temporal events and their relations are major challenges in natural language processing. Processing text on a sentence-by-sentence or expression-by-expression basis often fails, in part due to the challenge of capturing the global consistency of the text. We present an ensemble method, which reconciles the outputs of multiple classifiers of temporal expressions across the text using integer programming. Computational experiments show that the ensemble improves upon the best individual results from two recent challenges, SemEval-2013 TempEval-3 (Temporal Anno…
In this paper we introduce an evolutionary algorithm for the solution of linear integer programs. The strategy is based on the separation of the variables into the integer subset and the continuous subset; the integer variables are fixed by the evolutionary system, and the continuous ones are determined in function of them, by a linear program solver. We report results obtained for some standard benchmark problems, and compare them with those obtained by branch-and-bound. The performance of the evolutionary algorithm is promising. Good feasible solutions were generally obtained, and in some of…
We consider discrete bilevel optimization problems where the follower solves an integer program with a fixed number of variables. Using recent results in parametric integer programming, we present polynomial time algorithms for pure and mixed integer bilevel problems. For the mixed integer case where the leader's variables are continuous, our algorithm also detects whether the infimum cost fails to be attained, a difficulty that has been identified but not directly addressed in the literature. In this case it yields a ``better than fully polynomial time'' approximation scheme with running time…
We propose a trust-region method that solves a sequence of linear integer programs to tackle integer optimal control problems regularized with a total variation penalty. The total variation penalty allows us to prove the existence of minimizers of the integer optimal control problem. We introduce a local optimality concept for the problem, which arises from the infinite-dimensional perspective. In the case of a one-dimensional domain of the control function, we prove convergence of the iterates produced by our algorithm to points that satisfy first-order stationarity conditions for local optim…
We give an explicit geometric way to build mixed-integer programming (MIP) formulations for unions of polyhedra. The construction is simply described in terms of spanning hyperplanes in an r-dimensional linear space. The resulting MIP formulation is ideal, and uses exactly r integer variables and 2 x (# of spanning hyperplanes) general inequality constraints. We use this result to derive novel logarithmic-sized ideal MIP formulations for discontinuous piecewise linear functions and structures appearing in robotics and power systems problems.
We determine the maximal gap between the optimal values of an integer program and its linear programming relaxation, where the matrix and cost function are fixed but the right hand side is unspecified. Our formula involves irreducible decomposition of monomial ideals. The gap can be computed in polynomial time when the dimension is fixed.
It is well known that recurrent sandpile configurations can be characterized as the optimal solution of certain optimization problems. In this article, we present two new integer linear programming models, one that computes recurrent configurations and other that computes the order of the configuration. Finally, by using duality of linear programming, we are able to compute the identity configuration for the cone of a regular graph.
Two-stage stochastic mixed-integer programming (SMIP) problems with general integer variables in the second-stage are generally difficult to solve. This paper develops the theory of integer set reduction for characterizing the subset of the convex hull of feasible integer points of the second-stage subproblem which can be used for solving the SMIP. The basic idea is to consider a small enough subset of feasible integer points that is necessary for generating a valid inequality for the integer subproblem. An algorithm for obtaining such a subset based on the solution of the subproblem LP-relaxa…
In this work, we present a modeling framework for minimax mixed 0-1 fuzzy linear problems. It is based on extending the usual rewriting of crisp minimax problems via auxiliary variables to model the maximum of a finite set of fuzzy linear functions. We establish that the considered problem can be equivalently formulated as a multiple objective mixed integer programming problem. The framework is applied to a fully fuzzy version of the capacitated center facility location problem.
We consider integer programming and the semi-group membership problem. We provide the following theorem of the alternative: the system Ax=b has no nonnegative integral solution x if and only if p(b) <0 for some given polynomial p whose vector of coefficients lies in a convex cone that we characterize. We also provide a hierarchy of linear programming relaxations, where the continuous case Ax=b with x real and nonnegative, describes the first relaxation in the hierarchy.
We address the Interval Data Min-Max Regret 0-1 Integer Linear Programming problem (MMR-ILP), a variant of the 0-1 Integer Linear Programming problem where the objective function coefficients are uncertain. We solve MMR-ILP using a Benders-like Decomposition Algorithm and two metaheuristics for min-max regret problems with interval data. Computational experiments developed on variations of MIPLIB instances show that the heuristics obtain good results in a reasonable computational time when compared to the Benders-like Decomposition algorithm.
Designing efficient algorithms to compute Nash equilibria poses considerable challenges in Algorithmic Game Theory and Optimization. In this work, we employ integer programming techniques to compute Nash equilibria in Integer Programming Games, a class of simultaneous and non-cooperative games where each player solves a parametrized integer program. We introduce ZERO Regrets, a general and efficient cutting plane algorithm to compute, enumerate, and select Nash equilibria. Our framework leverages the concept of equilibrium inequality, an inequality valid for any Nash equilibrium, and the assoc…
Motivated by their success in the single-objective domain, we propose a very simple linear programming-based matheuristic for tri-objective binary integer programming. To tackle the problem, we obtain lower bound sets by means of the vector linear programming solver Bensolve. Then, simple heuristic approaches, such as rounding and path relinking, are applied to this lower bound set to obtain high-quality approximations of the optimal set of trade-off solutions. The proposed algorithm is compared to a recently suggested algorithm which is, to the best of our knowledge, the only existing matheur…
A standard quadratic program is an optimization problem that consists of minimizing a (nonconvex) quadratic form over the unit simplex. We focus on reformulating a standard quadratic program as a mixed integer linear programming problem. We propose two alternative mixed integer linear programming formulations. Our first formulation is based on casting a standard quadratic program as a linear program with complementarity constraints. We then employ binary variables to linearize the complementarity constraints. For the second formulation, we first derive an overestimating function of the objecti…
This paper presents algorithms for solving multiobjective integer programming problems. The algorithm uses Barvinok's rational functions of the polytope that defines the feasible region and provides as output the entire set of nondominated solutions for the problem. Theoretical complexity results on the algorithm are provided in the paper. Specifically, we prove that encoding the entire set of nondominated solutions of the problem is polynomially doable, when the dimension of the decision space is fixed. In addition, we provide polynomial delay algorithms for enumerating this set. An implement…
This paper addresses the single-item single-stocking location stochastic lot sizing problem under the $(s, S) $ policy. We first present a mixed integer non-linear programming (MINLP) formulation for determining near-optimal $(s, S)$ policy parameters. To tackle larger instances, we then combine the previously introduced MINLP model and a binary search approach. These models can be reformulated as mixed integer linear programming (MILP) models which can be easily implemented and solved by using off-the-shelf optimisation software. Computational experiments demonstrate that optimality gaps of t…
DENSO CORPORATION（电装公司）；National Institute of Advanced Industrial Science and Technology (AIST)（日本产业技术综合研究所）；DENSO IT Laboratory（电装IT实验室）
In edge computing, suppressing data size is a challenge for machine learning models that perform complex tasks such as autonomous driving, in which computational resources (speed, memory size and power) are limited. Efficient lossy compression of matrix data has been introduced by decomposing it into the product of an integer and real matrices. However, its optimisation is difficult as it requires simultaneous optimisation of an integer and real variables. In this paper, we improve this optimisation by utilising recently developed black-box optimisation (BBO) algorithms with an Ising solver fo…
We consider the constrained assortment optimization problem under the mixed multinomial logit model. Even moderately sized instances of this problem are challenging to solve directly using standard mixed-integer linear optimization formulations. This has motivated recent research exploring customized optimization strategies and approximation techniques. In contrast, we develop a novel conic quadratic mixed-integer formulation. This new formulation, together with McCormick inequalities exploiting the capacity constraints, enables the solution of large instances using commercial optimization sof…
We study the generalization of split, k-branch split, and intersection cuts from Mixed Integer Linear Programming to the realm of Mixed Integer Nonlinear Programming. Constructing such cuts requires calculating the convex hull of the difference between a convex set and an open set with a simple geometric structure. We introduce two techniques to give precise characterizations of such convex hulls and use them to construct split, k-branch split, and intersection cuts for several classes of non-polyhedral sets. In particular, we give simple formulas for split cuts for essentially all convex sets…
We consider so-called squaring the square-puzzles where a given square (or rectangle) should be dissected into smaller squares. For a specific instance of such problems we demonstrate that a mathematically rigorous solution can be quite involved. As an alternative to exhaustive enumeration using tailored algorithms we describe the general approach of formulating the problem as an integer linear program.
We propose a feasible active set method for convex quadratic programming problems with non-negativity constraints. This method is specifically designed to be embedded into a branch-and-bound algorithm for convex quadratic mixed integer programming problems. The branch-and-bound algorithm generalizes the approach for unconstrained convex quadratic integer programming proposed by Buchheim, Caprara and Lodi to the presence of linear constraints. The main feature of the latter approach consists in a sophisticated preprocessing phase, leading to a fast enumeration of the branch-and-bound nodes. Mor…
In this paper, we present FPT-algorithms for special cases of the shortest vector problem (SVP) and the integer linear programming problem (ILP), when matrices included to the problems' formulations are near square. The main parameter is the maximal absolute value of rank minors of matrices included to the problem formulation. Additionally, we present FPT-algorithms with respect to the same main parameter for the problems, when the matrices have no singular rank sub-matrices.
We consider integer and linear programming problems for which the linear constraints exhibit a (recursive) block-structure: The problem decomposes into independent and efficiently solvable sub-problems if a small number of constraints is deleted. A prominent example are $n$-fold integer programming problems and their generalizations which have received considerable attention in the recent literature. The previously known algorithms for these problems are based on the augmentation framework, a tailored integer programming variant of local search. In this paper we propose a different approach. O…
The Graphical Traveling Salesperson Problem (GTSP) is the problem of assigning, for a given weighted graph, a nonnegative number $x_e$ each edge $e$ such that the induced multi-subgraph is of minimum weight among those that are spanning, connected and Eulerian. Naturally, known mixed-integer programming formulations use integer variables $x_e$ in addition to others. Denis Naddef posed the challenge of finding a (reasonably simple) mixed-integer programming formulation that has integrality constraints only on these edge variables. Recently, Carr and Simonetti (IPCO 2021) showed that such a form…
We consider the integer points in a unimodular cone K ordered by a lexicographic rule defined by a lattice basis. To each integer point x in K we associate a family of inequalities (lex-cuts) that defines the convex hull of the integer points in K that are not lexicographically smaller than x. The family of lex-cuts contains the Chvatal-Gomory cuts, but does not contain and is not contained in the family of split cuts. This provides a finite cutting plane method to solve the integer program min{cx : x \in S \cap Z^n }, where S \subset R^n is a compact set and c \in Z^n . We analyze the number …
The paper provides a connection between Commutative Algebra and Integer Programming and contains two parts. The first one is devoted to the asymptotic behavior of integer programs with a fixed cost linear functional and the constraint sets consisting of a finite system of linear equations or inequalities with integer coefficients depending linearly on $n$. An integer $N_*$ is determined such that the optima of these integer programs are a quasi-linear function of $n$ for all $n\ge N_*$. Using results in the first part, one can bound in the second part the indices of stability of the Castelnuov…
N-fold integer programming is a fundamental problem with a variety of natural applications in operations research and statistics. Moreover, it is universal and provides a new, variable-dimension, parametrization of all of integer programming. The fastest algorithm for $n$-fold integer programming predating the present article runs in time $O(n^{g(A)}L)$ with $L$ the binary length of the numerical part of the input and $g(A)$ the so-called Graver complexity of the bimatrix $A$ defining the system. In this article we provide a drastic improvement and establish an algorithm which runs in time $O(…
In this article we study a broad class of integer programming problems in variable dimension. We show that these so-termed {\em n-fold integer programming problems} are polynomial time solvable. Our proof involves two heavy ingredients discovered recently: the equivalence of linear optimization and so-called directed augmentation, and the stabilization of certain Graver bases. We discuss several applications of our algorithm to multiway transportation problems and to packing problems. One important consequence of our results is a polynomial time algorithm for the $d$-dimensional integer transp…
