Submodular and fractionally subadditive (or equivalently XOS) functions play a fundamental role in combinatorial optimization, algorithmic game theory and machine learning. Motivated by learnability of these classes of functions from random examples, we consider the question of how well such functions can be approximated by low-degree polynomials in $\ell_2$ norm over the uniform distribution. This question is equivalent to understanding of the concentration of Fourier weight on low-degree coefficients, a central concept in Fourier analysis. We show that
1. For any submodular function $f:\{0,1\}^n \rightarrow [0,1]$, there is a polynomial of degree $O(\log (1/ε) / ε^{4/5})$ approximating $f$ within $ε$ in $\ell_2$, and there is a submodular function that requires degree $Ω(1/ε^{4/5})$.
2. For any XOS function $f:\{0,1\}^n \rightarrow [0,1]$, there is a polynomial of degree $O(1/ε)$ and there exists an XOS function that requires degree $Ω(1/ε)$.
This improves on previous approaches that all showed an upper bound of $O(1/ε^2)$ for submodular and XOS functions. The best previous lower bound was $Ω(1/ε^{2/3})$ for monotone submodular functions. Our techniques reveal new structural properties of submodular and XOS functions and the upper bounds lead to nearly optimal PAC learning algorithms for these classes of functions.
We investigate the approximability of several classes of real-valued functions by functions of a small number of variables ({\em juntas}). Our main results are tight bounds on the number of variables required to approximate a function $f:\{0,1\}^n \rightarrow [0,1]$ within $\ell_2$-error $ε$ over the uniform distribution: 1. If $f$ is submodular, then it is $ε$-close to a function of $O(\frac{1}{ε^2} \log \frac{1}ε)$ variables. This is an exponential improvement over previously known results. We note that $Ω(\frac{1}{ε^2})$ variables are necessary even for linear functions. 2. If $f$ is fracti…
Suppose we are given a submodular function $f$ over a set of elements, and we want to maximize its value subject to certain constraints. Good approximation algorithms are known for such problems under both monotone and non-monotone submodular functions. We consider these problems in a stochastic setting, where elements are not all active and we can only get value from active elements. Each element $e$ is active independently with some known probability $p_e$, but we don't know the element's status \emph{a priori}. We find it out only when we \emph{probe} the element $e$---probing reveals wheth…
In this paper, we study fundamental problems of maximizing DR-submodular continuous functions that have real-world applications in the domain of machine learning, economics, operations research and communication systems. It captures a subclass of non-convex optimization that provides both theoretical and practical guarantees. Here, we focus on minimizing regret for online arriving non-monotone DR-submodular functions over different types of convex sets: hypercube, down-closed and general convex sets. First, we present an online algorithm that achieves a $1/e$-approximation ratio with the regre…
Submodular functions are relevant to machine learning for at least two reasons: (1) some problems may be expressed directly as the optimization of submodular functions and (2) the lovasz extension of submodular functions provides a useful set of regularization functions for supervised and unsupervised learning. In this monograph, we present the theory of submodular functions from a convex analysis perspective, presenting tight links between certain polyhedra, combinatorial optimization and convex optimization problems. In particular, we show how submodular function minimization is equivalent t…
A set function is called XOS if it can be represented by the maximum of additive functions. When such a representation is fixed, the number of additive functions required to define the XOS function is called the width. In this paper, we study the problem of maximizing XOS functions in the value oracle model. The problem is trivial for the XOS functions of width $1$ because they are just additive, but it is already nontrivial even when the width is restricted to $2$. We show two types of tight bounds on the polynomial-time approximability for this problem. First, in general, the approximation b…
Near-Optimal Online Learning for Multi-Agent Submodular Coordination: Tight Approximation and Communication Efficiency
Nanyang Technological University（南洋理工大学）；City University of Hong Kong（香港城市大学）；Sun Yat-sen University（中山大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
INRIA（法国国家信息与自动化研究所）；Laboratoire d’Informatique de l’Ecole Normale Supérieure（巴黎高等师范学院计算机实验室）
Sparse methods for supervised learning aim at finding good linear predictors from as few variables as possible, i.e., with small cardinality of their supports. This combinatorial selection problem is often turned into a convex optimization problem by replacing the cardinality function by its convex envelope (tightest convex lower bound), in this case the L1-norm. In this paper, we investigate more general set-functions than the cardinality, that may incorporate prior knowledge or structural constraints which are common in many applications: namely, we show that for nondecreasing submodular set…
Greedy algorithms are widely used for problems in machine learning such as feature selection and set function optimization. Unfortunately, for large datasets, the running time of even greedy algorithms can be quite high. This is because for each greedy step we need to refit a model or calculate a function using the previously selected choices and the new candidate. Two algorithms that are faster approximations to the greedy forward selection were introduced recently ([Mirzasoleiman et al. 2013, 2015]). They achieve better performance by exploiting distributed computation and stochastic evaluat…
We show that for any constant $ε> 0$ and $p \ge 1$, it is possible to distinguish functions $f : \{0,1\}^n \to [0,1]$ that are submodular from those that are $ε$-far from every submodular function in $\ell_p$ distance with a constant number of queries. More generally, we extend the testing-by-implicit-learning framework of Diakonikolas et al. (2007) to show that every property of real-valued functions that is well-approximated in $\ell_2$ distance by a class of $k$-juntas for some $k = O(1)$ can be tested in the $\ell_p$-testing model with a constant number of queries. This result, combined wi…
In combinatorial auctions, a designer must decide how to allocate a set of indivisible items amongst a set of bidders. Each bidder has a valuation function which gives the utility they obtain from any subset of the items. Our focus is specifically on welfare maximization, where the objective is to maximize the sum of valuations that the bidders place on the items that they were allocated (the valuation functions are assumed to be reported truthfully). We analyze an online problem in which the algorithm is not given the set of bidders in advance. Instead, the bidders are revealed sequentially i…
In this paper we study the approximate learnability of valuations commonly used throughout economics and game theory for the quantitative encoding of agent preferences. We provide upper and lower bounds regarding the learnability of important subclasses of valuation functions that express no-complementarities. Our main results concern their approximate learnability in the distributional learning (PAC-style) setting. We provide nearly tight lower and upper bounds of $\tildeΘ(n^{1/2})$ on the approximation factor for learning XOS and subadditive valuations, both widely studied superclasses of su…
Yale Institute for Network Science, Yale University（耶鲁大学耶鲁网络科学研究所）；The Open University of Israel（以色列开放大学）
In this paper, we consider the unconstrained submodular maximization problem. We propose the first algorithm for this problem that achieves a tight $(1/2-\varepsilon)$-approximation guarantee using $\tilde{O}(\varepsilon^{-1})$ adaptive rounds and a linear number of function evaluations. No previously known algorithm for this problem achieves an approximation ratio better than $1/3$ using less than $Ω(n)$ rounds of adaptivity, where $n$ is the size of the ground set. Moreover, our algorithm easily extends to the maximization of a non-negative continuous DR-submodular function subject to a box …
University of Pennsylvania（宾夕法尼亚大学）；Yale University（耶鲁大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
In this paper, we consider the general non-oblivious stochastic optimization where the underlying stochasticity may change during the optimization procedure and depends on the point at which the function is evaluated. We develop Stochastic Frank-Wolfe++ ($\text{SFW}{++} $), an efficient variant of the conditional gradient method for minimizing a smooth non-convex function subject to a convex body constraint. We show that $\text{SFW}{++} $ converges to an $ε$-first order stationary point by using $O(1/ε^3)$ stochastic gradients. Once further structures are present, $\text{SFW}{++}$'s theoretica…
We present a polynomial time algorithm for online maximization of $k$-submodular maximization. For online (nonmonotone) $k$-submodular maximization, our algorithm achieves a tight approximate factor in an approximate regret. For online monotone $k$-submodular maximization, our approximate-regret matches to the best-known approximation ratio, which is tight asymptotically as $k$ tends to infinity. Our approach is based on the Blackwell approachability theorem and online linear optimization.
In this paper, we propose an unifying view of several recently proposed structured sparsity-inducing norms. We consider the situation of a model simultaneously (a) penalized by a set- function de ned on the support of the unknown parameter vector which represents prior knowledge on supports, and (b) regularized in Lp-norm. We show that the natural combinatorial optimization problems obtained may be relaxed into convex optimization problems and introduce a notion, the lower combinatorial envelope of a set-function, that characterizes the tightness of our relaxations. We moreover establish links…
Streaming algorithms are generally judged by the quality of their solution, memory footprint, and computational complexity. In this paper, we study the problem of maximizing a monotone submodular function in the streaming setting with a cardinality constraint $k$. We first propose Sieve-Streaming++, which requires just one pass over the data, keeps only $O(k)$ elements and achieves the tight $(1/2)$-approximation guarantee. The best previously known streaming algorithms either achieve a suboptimal $(1/4)$-approximation with $Θ(k)$ memory or the optimal $(1/2)$-approximation with $O(k\log k)$ m…
Submodular functions are a fundamental object of study in combinatorial optimization, economics, machine learning, etc. and exhibit a rich combinatorial structure. Many subclasses of submodular functions have also been well studied and these subclasses widely vary in their complexity. Our motivation is to understand the relative complexity of these classes of functions. Towards this, we consider the question of how well can one class of submodular functions be approximated by another (simpler) class of submodular functions. Such approximations naturally allow algorithms designed for the simple…
University of Southern California（南加州大学）；Rensselaer Polytechnic Institute（伦斯勒理工学院）
We consider the problem of maximizing non-negative non-decreasing set functions. Although most of the recent work focus on exploiting submodularity, it turns out that several objectives we encounter in practice are not submodular. Nonetheless, often we leverage the greedy algorithms used in submodular functions to determine a solution to the non-submodular functions. Hereafter, we propose to address the original problem by \emph{approximating} the non-submodular function and analyze the incurred error, as well as the performance trade-offs. To quantify the approximation error, we introduce a n…
We consider the problem of maximizing a monotone submodular function under a knapsack constraint. We show that, for any fixed $ε> 0$, there exists a polynomial-time algorithm with an approximation ratio $1-c/e-ε$, where $c \in [0,1]$ is the (total) curvature of the input function. This approximation ratio is tight up to $ε$ for any $c \in [0,1]$. To the best of our knowledge, this is the first result for a knapsack constraint that incorporates the curvature to obtain an approximation ratio better than $1-1/e$, which is tight for general submodular functions. As an application of our result, we…
We present an optimal, combinatorial 1-1/e approximation algorithm for monotone submodular optimization over a matroid constraint. Compared to the continuous greedy algorithm (Calinescu, Chekuri, Pal and Vondrak, 2008), our algorithm is extremely simple and requires no rounding. It consists of the greedy algorithm followed by local search. Both phases are run not on the actual objective function, but on a related non-oblivious potential function, which is also monotone submodular. Our algorithm runs in randomized time O(n^8u), where n is the rank of the given matroid and u is the size of its g…
Motivated by applications in machine learning, such as subset selection and data summarization, we consider the problem of maximizing a monotone submodular function subject to mixed packing and covering constraints. We present a tight approximation algorithm that for any constant $ε>0$ achieves a guarantee of $1-\frac{1}{\mathrm{e}}-ε$ while violating only the covering constraints by a multiplicative factor of $1-ε$. Our algorithm is based on a novel enumeration method, which unlike previous known enumeration techniques, can handle both packing and covering constraints. We extend the above mai…
Seminal work by Edmonds and Lovasz shows the strong connection between submodularity and convexity. Submodular functions have tight modular lower bounds, and subdifferentials in a manner akin to convex functions. They also admit poly-time algorithms for minimization and satisfy the Fenchel duality theorem and the Discrete Seperation Theorem, both of which are fundamental characteristics of convex functions. Submodular functions also show signs similar to concavity. Submodular maximization, though NP hard, admits constant factor approximation guarantees. Concave functions composed with modular …
This paper presents a polynomial-time $1/2$-approximation algorithm for maximizing nonnegative $k$-submodular functions. This improves upon the previous $\max\{1/3, 1/(1+a)\}$-approximation by Ward and Živný~(SODA'14), where $a=\max\{1, \sqrt{(k-1)/4}\}$. We also show that for monotone $k$-submodular functions there is a polynomial-time $k/(2k-1)$-approximation algorithm while for any $\varepsilon>0$ a $((k+1)/2k+\varepsilon)$-approximation algorithm for maximizing monotone $k$-submodular functions would require exponentially many queries. In particular, our hardness result implies that our al…
Simon Fraser University（西蒙菲莎大学）；National Institute of Informatics（信息科学研究所）
The problem of maximizing nonnegative monotone submodular functions under a certain constraint has been intensively studied in the last decade, and a wide range of efficient approximation algorithms have been developed for this problem. Many machine learning problems, including data summarization and influence maximization, can be naturally modeled as the problem of maximizing monotone submodular functions. However, when such applications involve sensitive data about individuals, their privacy concerns should be addressed. In this paper, we study the problem of maximizing monotone submodular f…
We consider non-monotone DR-submodular function maximization, where DR-submodularity (diminishing return submodularity) is an extension of submodularity for functions over the integer lattice based on the concept of the diminishing return property. Maximizing non-monotone DR-submodular functions has many applications in machine learning that cannot be captured by submodular set functions. In this paper, we present a $\frac{1}{2+ε}$-approximation algorithm with a running time of roughly $O(\frac{n}ε\log^2 B)$, where $n$ is the size of the ground set, $B$ is the maximum value of a coordinate, an…
A deterministic approximation algorithm is presented for the maximization of non-monotone submodular functions over a ground set of size $n$ subject to cardinality constraint $k$; the algorithm is based upon the idea of interlacing two greedy procedures. The algorithm uses interlaced, thresholded greedy procedures to obtain tight ratio $1/4 - ε$ in $O \left( \frac{n}ε \log \left( \frac{k}ε \right) \right)$ queries of the objective function, which improves upon both the ratio and the quadratic time complexity of the previously fastest deterministic algorithm for this problem. The algorithm is v…
Prophet inequalities and secretary problems have been extensively studied in recent years due to their elegance, connections to online algorithms, stochastic optimization, and mechanism design problems in game theoretic settings. Rubinstein and Singla developed a notion of combinatorial prophet inequalities in order to generalize the standard prophet inequality setting to combinatorial valuation functions such as submodular and subadditive functions. For non-negative submodular functions they demonstrated a constant factor prophet inequality for matroid constraints. Along the way they showed a…
NYU Tandon School of Engineering（纽约大学坦登工程学院）；Brooklyn College, City University of New York（纽约城市大学布鲁克林学院）；IBM T. J. Watson Research Center（IBM托马斯·J·沃森研究中心）
We show that the Adaptive Greedy algorithm of Golovin and Krause (2011) achieves an approximation bound of $(\ln (Q/η)+1)$ for Stochastic Submodular Cover: here $Q$ is the "goal value" and $η$ is the smallest non-zero marginal increase in utility deliverable by an item. (For integer-valued utility functions, we show a bound of $H(Q)$, where $H(Q)$ is the $Q^{th}$ Harmonic number.) Although this bound was claimed by Golovin and Krause in the original version of their paper, the proof was later shown to be incorrect by Nan and Saligrama (2017). The subsequent corrected proof of Golovin and Kraus…
Yale Institute for Network Science（耶鲁网络科学研究所）；Yale University（耶鲁大学）；University of Pennsylvania（宾夕法尼亚大学）
In this paper, we consider the problem of black box continuous submodular maximization where we only have access to the function values and no information about the derivatives is provided. For a monotone and continuous DR-submodular function, and subject to a bounded convex body constraint, we propose Black-box Continuous Greedy, a derivative-free algorithm that provably achieves the tight $[(1-1/e)OPT-ε]$ approximation guarantee with $O(d/ε^3)$ function evaluations. We then extend our result to the stochastic setting where function values are subject to stochastic zero-mean noise. It is thro…
We investigate two new optimization problems -- minimizing a submodular function subject to a submodular lower bound constraint (submodular cover) and maximizing a submodular function subject to a submodular upper bound constraint (submodular knapsack). We are motivated by a number of real-world applications in machine learning including sensor placement and data subset selection, which require maximizing a certain submodular function (like coverage or diversity) while simultaneously minimizing another (like cooperative cost). These problems are often posed as minimizing the difference between…
Google Brain（谷歌大脑）；Carnegie Mellon University（卡内基梅隆大学）；Stanford University（斯坦福大学）
We study the complexity of learning and approximation of self-bounding functions over the uniform distribution on the Boolean hypercube ${0,1}^n$. Informally, a function $f:{0,1}^n \rightarrow \mathbb{R}$ is self-bounding if for every $x \in {0,1}^n$, $f(x)$ upper bounds the sum of all the $n$ marginal decreases in the value of the function at $x$. Self-bounding functions include such well-known classes of functions as submodular and fractionally-subadditive (XOS) functions. They were introduced by Boucheron et al. (2000) in the context of concentration of measure inequalities. Our main result…
We investigate the performance of the standard Greedy algorithm for cardinality constrained maximization of non-submodular nondecreasing set functions. While there are strong theoretical guarantees on the performance of Greedy for maximizing submodular functions, there are few guarantees for non-submodular ones. However, Greedy enjoys strong empirical performance for many important non-submodular functions, e.g., the Bayesian A-optimality objective in experimental design. We prove theoretical guarantees supporting the empirical performance. Our guarantees are characterized by a combination of …
We consider the classical problem of maximizing a monotone submodular function subject to a cardinality constraint, which, due to its numerous applications, has recently been studied in various computational models. We consider a clean multi-player model that lies between the offline and streaming model, and study it under the aspect of one-way communication complexity. Our model captures the streaming setting (by considering a large number of players), and, in addition, two player approximation results for it translate into the robust setting. We present tight one-way communication complexity…
We propose and analyze batch greedy heuristics for cardinality constrained maximization of non-submodular non-decreasing set functions. We consider the standard greedy paradigm, along with its distributed greedy and stochastic greedy variants. Our theoretical guarantees are characterized by the combination of submodularity and supermodularity ratios. We argue how these parameters define tight modular bounds based on incremental gains, and provide a novel reinterpretation of the classical greedy algorithm using the minorize-maximize (MM) principle. Based on that analogy, we propose a new class …
In machine learning and big data, the optimization objectives based on set-cover, entropy, diversity, influence, feature selection, etc. are commonly modeled as submodular functions. Submodular (function) maximization is generally NP-hard, even in the absence of constraints. Recently, submodular maximization has been successfully investigated for the settings where the objective function is monotone or the constraint is computation-tractable. However, maximizing nonmonotone submodular function with complex constraints is not yet well-understood. In this paper, we consider the nonmonotone submo…
The greedy algorithm for monotone submodular function maximization subject to cardinality constraint is guaranteed to approximate the optimal solution to within a $1-1/e$ factor. Although it is well known that this guarantee is essentially tight in the worst case -- for greedy and in fact any efficient algorithm, experiments show that greedy performs better in practice. We observe that for many applications in practice, the empirical distribution of the budgets (i.e., cardinality constraints) is supported on a wide range, and moreover, all the existing hardness results in theory break under a …
A $k$-submodular function is an extension of a submodular function in that its input is given by $k$ disjoint subsets instead of a single subset. For unconstrained nonnegative $k$-submodular maximization, Ward and Živný proposed a constant-factor approximation algorithm, which was improved by the recent work of Iwata, Tanigawa and Yoshida presenting a $1/2$-approximation algorithm. Iwata et al. also provided a $k/(2k-1)$-approximation algorithm for monotone $k$-submodular maximization and proved that its approximation ratio is asymptotically tight. More recently, Ohsaka and Yoshida proposed co…
Google Research（谷歌研究院）；MIT（麻省理工学院）；UC Berkeley（加州大学伯克利分校）
We give a deterministic nearly-linear time algorithm for approximating any point inside a convex polytope with a sparse convex combination of the polytope's vertices. Our result provides a constructive proof for the Approximate Carathéodory Problem, which states that any point inside a polytope contained in the $\ell_p$ ball of radius $D$ can be approximated to within $ε$ in $\ell_p$ norm by a convex combination of only $O\left(D^2 p/ε^2\right)$ vertices of the polytope for $p \geq 2$. We also show that this bound is tight, using an argument based on anti-concentration for the binomial distrib…
We study the problem of Regularized Unconstrained Submodular Maximization (RegularizedUSM) as defined by Bodek and Feldman [BF22]. In this problem, you are given a non-monotone non-negative submodular function $f:2^{\mathcal N}\to \mathbb R_{\ge 0}$ and a linear function $\ell:2^{\mathcal N}\to \mathbb R$ over the same ground set $\mathcal N$, and the objective is to output a set $T\subseteq \mathcal N$ approximately maximizing the sum $f(T)+\ell(T)$. Specifically, an algorithm is said to provide an $(α,β)$-approximation for RegularizedUSM if it outputs a set $T$ such that $\mathbb E[f(T)+\ell…
Submodular functions are an important class of functions in combinatorial optimization which satisfy the natural properties of decreasing marginal costs. The study of these functions has led to strong structural properties with applications in many areas. Recently, there has been significant interest in extending the theory of algorithms for optimizing combinatorial problems (such as network design problem of spanning tree) over submodular functions. Unfortunately, the lower bounds under the general class of submodular functions are known to be very high for many of the classical problems. In …
We consider the following general network design problem on directed graphs. The input is an asymmetric metric $(V,c)$, root $r^{*}\in V$, monotone submodular function $f:2^V\rightarrow \mathbb{R}_+$ and budget $B$. The goal is to find an $r^{*}$-rooted arborescence $T$ of cost at most $B$ that maximizes $f(T)$. Our main result is a simple quasi-polynomial time $O(\frac{\log k}{\log\log k})$-approximation algorithm for this problem, where $k\le |V|$ is the number of vertices in an optimal solution. To the best of our knowledge, this is the first non-trivial approximation ratio for this problem…
We show that Set Cover on instances with $N$ elements cannot be approximated within $(1-γ)\ln N$-factor in time exp($N^{γ-δ})$, for any $0 < γ< 1$ and any $δ> 0$, assuming the Exponential Time Hypothesis. This essentially matches the best upper bound known by Cygan et al.\ (IPL, 2009) of $(1-γ)\ln N$-factor in time $exp(O(N^γ))$. The lower bound is obtained by extracting a standalone reduction from Label Cover to Set Cover from the work of Moshkovitz (Theory of Computing, 2015), and applying it to a different PCP theorem than done there. We also obtain a tighter lower bound when conditioning o…
National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）；University of Science and Technology of China（中国科学技术大学）；State University of New York at Buffalo（纽约州立大学布法罗分校）
Monotone submodular maximization with a knapsack constraint is NP-hard. Various approximation algorithms have been devised to address this optimization problem. In this paper, we revisit the widely known modified greedy algorithm. First, we show that this algorithm can achieve an approximation factor of $0.405$, which significantly improves the known factors of $0.357$ given by Wolsey and $(1-1/\mathrm{e})/2\approx 0.316$ given by Khuller et al. More importantly, our analysis closes a gap in Khuller et al.'s proof for the extensively mentioned approximation factor of $(1-1/\sqrt{\mathrm{e}})\a…
