arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.14987stat.MLcs.LG
收藏

关于平滑数据下经验风险最小化的性能研究

On the Performance of Empirical Risk Minimization with Smoothed Data

MIT(麻省理工学院);University of California, Berkeley(加利福尼亚大学伯克利分校)

Adam Block, Alexander Rakhlin, Abhishek Shetty

AI总结:

本文研究基准测度未知的平滑在线学习中经验风险最小化(ERM)的性能,证明其误差可达 $\tilde O(\sqrt{\mathrm{comp}(\mathcal F)\cdot T})$,并给出紧的下界,揭示平滑数据与 iid 数据下 ERM 的性能分离。

摘要:

为了规避序列决策中的统计与计算困难结果,近期工作考虑了平滑在线学习,其中每个时刻的数据分布在以历史为条件时,被假设相对于某个基准测度具有有界似然比。虽然此前的工作已经展示了平滑性的好处,但它们要么假设基准测度对学习者已知,要么提出了仅在特殊情况下适用且计算上低效的算法。本文研究了更一般的设定,即基准测度对学习者未知,特别关注当数据是良设定的且平滑时,使用平方损失的经验风险最小化(ERM)的性能。我们表明,在这一设定下,只要一个函数类能够用独立同分布(iid)数据学习,ERM 就能实现次线性误差;特别是,ERM 实现的误差规模为 $\tilde O( \sqrt{\mathrm{comp}(\mathcal F)\cdot T} )$,其中 $\mathrm{comp}(\mathcal F)$ 是用 iid 数据学习 $\mathcal F$ 的统计复杂度。在此过程中,我们证明了一个针对平滑数据的新范数比较界,它构成了适用于任意非线性函数类的、针对依赖数据的第一个精确范数比较结果。我们用一个下界补充了这些结果,该下界表明我们对 ERM 的分析本质上是紧的,从而确立了 ERM 在平滑数据与 iid 数据之间性能上的分离。

英文摘要:

In order to circumvent statistical and computational hardness results in sequential decision-making, recent work has considered smoothed online learning, where the distribution of data at each time is assumed to have bounded likeliehood ratio with respect to a base measure when conditioned on the history. While previous works have demonstrated the benefits of smoothness, they have either assumed that the base measure is known to the learner or have presented computationally inefficient algorithms applying only in special cases. This work investigates the more general setting where the base measure is \emph{unknown} to the learner, focusing in particular on the performance of Empirical Risk Minimization (ERM) with square loss when the data are well-specified and smooth. We show that in this setting, ERM is able to achieve sublinear error whenever a class is learnable with iid data; in particular, ERM achieves error scaling as $\tilde O( \sqrt{\mathrm{comp}(\mathcal F)\cdot T} )$, where $\mathrm{comp}(\mathcal F)$ is the statistical complexity of learning $\mathcal F$ with iid data. In so doing, we prove a novel norm comparison bound for smoothed data that comprises the first sharp norm comparison for dependent data applying to arbitrary, nonlinear function classes. We complement these results with a lower bound indicating that our analysis of ERM is essentially tight, establishing a separation in the performance of ERM between smoothed and iid data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑