单调敌手学习的最优速率

中文摘要 AI 辅助

单调敌手观察一个独立同分布的带标签样本，并添加有限个其选择的额外样本，每个额外样本都被目标假设正确标记。学习者会看到组合样本的均匀洗牌版本，并在原始分布上进行评分。所有样本都有正确标签，但插入的样本依赖于干净样本，因此组合样本不具备可交换性。引入该模型的Larsen、Pabbaraju和Shetty表明，对于VC维为d的类，经验风险最小化可达到预期误差O((d/n)log(n/d))，且所有已知的最优学习者都可能被推离PAC学习的最优速率Θ(d/n)。他们提出疑问：额外的对数项是这些特定算法的人为产物，还是缺乏可交换性的固有结果？我们证明，除VC维为1的情况外，这种额外代价是固有存在的。在VC维为d的类和已知有限插入预算的最坏情况下，极小极大预期误差在d=1时为Θ(1/n)，在d≥2时为Θ((d/n)log(n/d))。当用Littlestone维数d_L替代d时，同样的速率成立，因此干净的在线到批处理速率O(d_L/n)也无法实现。因此，有点反直觉的是，即使对于在线学习中具有有限错误界的类，添加正确标记的样本也会使学习难度增加一个对数因子。维数1的上界由一个简单的非适当学习者实现，其分析采用了基于单包含图的留一法论证。我们所有的下界都是基础的，来自单一构造：一个显式类和先验，其中两个目标假设在具有不可忽略质量的点上不同，却产生相同的样本。

英文摘要

A monotone adversary observes an i.i.d. labeled sample and appends a finite number of further examples of its choice, every one of them labeled correctly by the target hypothesis. The learner sees a uniform shuffle of the combined sample and is scored on the original distribution. Every example is correctly labeled, but the insertions depend on the clean sample, so the combined sample is not exchangeable. Larsen, Pabbaraju, and Shetty, who introduced this model, showed that empirical risk minimization attains expected error $O((d/n)\log(n/d))$ for classes of VC dimension $d$, and that every known optimal learner can be pushed away from the $Θ(d/n)$ rate, optimal for PAC learning. They asked whether the extra logarithm is an artifact of those particular algorithms or an inherent consequence of the lack of exchangeability. We show that this additional cost is inherent beyond VC dimension one. In the worst case over classes of VC dimension $d$ and over known finite insertion budgets, the minimax expected error is $Θ(1/n)$ at $d=1$ and $Θ((d/n)\log(n/d))$ for $d\geq 2$. The same rates hold with Littlestone dimension $d_{\mathrm L}$ in place of $d$, so the clean online-to-batch rate $O(d_{\mathrm L}/n)$ is unattainable as well. Thus, somewhat counterintuitively, adding correctly labeled examples can make learning harder by a logarithmic factor, even for classes that admit finite mistake bounds in online learning. The dimension-one upper bound is achieved by a simple improper learner whose analysis adapts the leave-one-out argument underlying the one-inclusion graph. All of our lower bounds are elementary and come from a single construction: an explicit class and prior on which two target hypothesis, which differ a point of nonnegligible mass, produce the same sample.

发表机构

• Stanford University（斯坦福大学）

机构由 AI 辅助整理，请以论文原文为准。