Structured prediction tasks in machine learning involve the simultaneous prediction of multiple labels. This is typically done by maximizing a score function on the space of labels, which decomposes as a sum of pairwise elements, each depending on two specific labels. Intuitively, the more pairwise terms are used, the better the expected accuracy. However, there is currently no theoretical account of this intuition. This paper takes a significant step in this direction.
We formulate the problem as classifying the vertices of a known graph $G=(V,E)$, where the vertices and edges of the graph are labelled and correlate semi-randomly with the ground truth. We show that the prospects for achieving low expected Hamming error depend on the structure of the graph $G$ in interesting ways. For example, if $G$ is a very poor expander, like a path, then large expected Hamming error is inevitable. Our main positive result shows that, for a wide class of graphs including 2D grid graphs common in machine vision applications, there is a polynomial-time algorithm with small and information-theoretically near-optimal expected error. Our results provide a first step toward a theoretical justification for the empirical success of the efficient approximate inference algorithms that are used for structured prediction in models where exact inference is intractable.
Tightening Mutual Information Based Bounds on Generalization Error
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Massachusetts Institute of Technology(麻省理工学院);University at Buffalo, The State University of New York(纽约州立大学布法罗分校)
Approximate Inference in Structured Instances with Noisy Categorical Observations
University of Waterloo(滑铁卢大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校)
我们研究了在噪声观测存在的情况下,恢复具有分类随机变量的结构化实例的潜在地面真值标记的问题。我们提出了一个新的近似算法,实现低汉明误差的存在噪声的顶点和边的观察与分类变量的图。我们的主要结果表明,对数依赖的汉明误差的类别的随机变量的数量。我们的方法与具有固定数量的聚类的相关聚类相连接。我们的研究结果将Globerson et al.(2015)和Foster et al.(2018)的工作推广到了分类标签的情况,他们研究了二元标签下结构化预测的难度。
Tight Generalization Error Bounds for Stochastic Gradient Descent in Non-convex Learning
School of Mathematics and Statistics(数学与统计学学院);Guangxi Normal University(广西师范大学);School of Mathematics(数学学院);Hohai University(河海大学);College of Marine Life Sciences(海洋生命科学学院);Ocean University of China(中国海洋大学);State Key Laboratory of Mathematical Sciences(数学科学国家重点实验室);Academy of Mathematics and Systems Science(数学与系统科学研究院);University of Chinese Academy of Sciences(中国科学院大学)
Almost Tight Error Bounds on Differentially Private Continual Counting
Faculty of Computer Science, University of Vienna(维也纳大学计算机学院);Rutgers University(罗格斯大学);Indian Statistical Institute, Delhi(印度统计研究所德里中心);Fujitsu Research of America(美国富士通研究所)
Byzantine-Robust Distributed SGD: A Unified Analysis and Tight Error Bounds
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学前沿交叉科学学院);School of Mathematical Sciences, University of Chinese Academy of Sciences(中国科学院大学数学科学学院);Ministry of Education Key Laboratory of Mathematics and Information Networks, School of Mathematical Sciences, Beijing University of Posts and Telecommunications(北京邮电大学数学科学学院数学与信息网络教育部重点实验室)
Agnostic Q-learning with Function Approximation in Deterministic Systems: Tight Bounds on Approximation Error and Sample Complexity
Institute for Advanced Study(高等研究院);Princeton University(普林斯顿大学);University of California, San Diego(加利福尼亚大学圣地亚哥分校);Carnegie Mellon University(卡内基梅隆大学)
本文研究确定性系统中具有函数逼近的不可知$Q$学习问题,其中最优$Q$函数可由类$\mathcal{F}$中的函数逼近,逼近误差为$δ\ge 0$ .我们提出了一种新的基于递归的算法,并表明,如果$δ= O\left(ρ/\sqrt{\dim_E}\right)$,那么可以找到最佳的政策使用$O\left(\dim_E\right)$轨迹,其中$ρ$是最佳的$Q$ -值的最佳行动和第二最佳行动和$\dim_E$之间的差距是$\mathcal{F}$的Eluder维。我们的结果有两个含义:1)结合[Du et al.,ICLR 2020],我们的上界表明条件$δ= \widetildeΘ\left(ρ/\sqrt{\mathrm{dim}_E}\right)$对于多项式样本复杂度的算法是必要和充分的. 2)结合[Wen and Van Roy,NIPS 2013]中的下限,我们的上限表明,即使在不可知论的设置中,样本复杂度$\widetildeΘ\left(\mathrm{dim}_E\right)$也是紧密的。因此,我们解决了[Wen and Van Roy,NIPS 2013]中提出的不可知$Q$学习的开放问题。我们进一步扩展我们的算法的随机奖励设置,并获得类似的结果。
Segmentation Rectification for Video Cutout via One-Class Structured Learning
Doheny Eye Institute(多赫尼眼科研究所);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Singapore University of Technology and Design(新加坡科技设计大学);Adobe Research(奥多比研究院);Zhejiang University(浙江大学)
Private Gradient Descent for Linear Regression: Tighter Error Bounds and Instance-Specific Uncertainty Estimation
Paul G Allen School of Computer Science and Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院);Boston University(波士顿大学);Google DeepMind(谷歌DeepMind)