arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2011.14126cs.LGmath.STstat.MLstat.TH
收藏

统计学习中的风险单调性

Risk-Monotonicity in Statistical Learning

Australian National University(澳大利亚国立大学);Massachusetts Institute of Technology(麻省理工学院)

Zakaria Mhammedi

摘要:

在机器学习的许多应用中,数据的获取是一项艰巨的任务,人们希望并期望群体风险随着数据点的增加而单调降低(更好的性能)是很自然的。令人惊讶的是,即使是最标准的算法,即使是最小化经验风险的算法,情况也并非如此。训练中的风险和不稳定性的非单调行为已经表现出来,并出现在流行的深度学习范式双下降的描述下。这些问题凸显了当前对学习算法的理解和泛化能力的不足。因此,至关重要的是要继续关注这一问题,并对这种行为进行定性。在本文中,我们在弱假设下推导出了一般统计学习设置的第一个一致和风险单调(高概率)算法,从而回答了Viering等人2019年提出的关于如何避免风险曲线非单调行为的一些问题。我们进一步表明,风险单调性不一定是以更坏的超额风险率为代价的。为了实现这一点,我们得到了新的经验Bernstein型浓度不等式的独立利益,保持一定的非独立同分布~鞅差序列(Martingale Difference Sequences)

英文摘要:

Acquisition of data is a difficult task in many applications of machine learning, and it is only natural that one hopes and expects the population risk to decrease (better performance) monotonically with increasing data points. It turns out, somewhat surprisingly, that this is not the case even for the most standard algorithms that minimize the empirical risk. Non-monotonic behavior of the risk and instability in training have manifested and appeared in the popular deep learning paradigm under the description of double descent. These problems highlight the current lack of understanding of learning algorithms and generalization. It is, therefore, crucial to pursue this concern and provide a characterization of such behavior. In this paper, we derive the first consistent and risk-monotonic (in high probability) algorithms for a general statistical learning setting under weak assumptions, consequently answering some questions posed by Viering et al. 2019 on how to avoid non-monotonic behavior of risk curves. We further show that risk monotonicity need not necessarily come at the price of worse excess risk rates. To achieve this, we derive new empirical Bernstein-like concentration inequalities of independent interest that hold for certain non-i.i.d.~processes such as Martingale Difference Sequences.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑