A seminal result in learning theory characterizes the PAC learnability of binary classes through the Vapnik-Chervonenkis dimension. Extending this characterization to the general multiclass setting has been open since the pioneering works on multiclass PAC learning in the late 1980s. This work resolves this problem: we characterize multiclass PAC learnability through the DS dimension, a combinatorial dimension defined by Daniely and Shalev-Shwartz (2014).
The classical characterization of the binary case boils down to empirical risk minimization. In contrast, our characterization of the multiclass case involves a variety of algorithmic ideas; these include a natural setting we call list PAC learning. In the list learning setting, instead of predicting a single outcome for a given unseen input, the goal is to provide a short menu of predictions.
Our second main result concerns the Natarajan dimension, which has been a central candidate for characterizing multiclass learnability. This dimension was introduced by Natarajan (1988) as a barrier for PAC learning. Whether the Natarajan dimension characterizes PAC learnability in general has been posed as an open question in several papers since. This work provides a negative answer: we construct a non-learnable class with Natarajan dimension one.
For the construction, we identify a fundamental connection between concept classes and topology (i.e., colorful simplicial complexes). We crucially rely on a deep and involved construction of hyperbolic pseudo-manifolds by Januszkiewicz and Swiatkowski. It is interesting that hyperbolicity is directly related to learning problems that are difficult to solve although no obvious barriers exist. This is another demonstration of the fruitful links machine learning has with different areas in mathematics.
The Sample Complexity of Multiclass and Sparse Contextual Bandits
Tel Aviv University（特拉维夫大学）；Massachusetts Institute of Technology（麻省理工学院）；Google Research Tel Aviv（谷歌研究特拉维夫）；Technion—Israel Institute of Technology（技术学院—以色列理工学院）
Institute for Data, Econometrics, Algorithms, and Learning (IDEAL), hosted by UIC and TTIC（数据、计量经济学、算法与学习研究所（IDEAL））；UIC（伊利诺伊大学芝加哥分校）；TTIC（丰田技术学院）；Purdue University（普渡大学）
