缓解多重下降:风险单调化的模型无关框架
Mitigating multiple descents: A model-agnostic framework for risk monotonization
摘要:
最近的经验和理论分析的几个常用的预测程序揭示了一个特殊的风险行为在高维,称为双/多下降,其中的渐近风险是一个非单调的功能或参数的数量限制纵横比的样本量的函数。为了减轻这种不良行为,我们开发了一个通用框架的风险单调交叉验证的基础上,作为输入的通用预测程序,并返回一个修改后的程序,其样本外的预测风险是渐进的,单调的限制纵横比。作为我们框架的一部分,我们提出了两种数据驱动的方法,即零步和一步,分别类似于装袋和提升,并表明,在非常温和的假设下,它们可证明实现单调渐近风险行为。我们的结果适用于各种各样的预测程序和损失函数,并不需要一个指定的(参数)模型。我们通过对最小$\ell_2$、$\ell_1$ -范数最小二乘预测程序的具体分析来举例说明我们的框架。作为我们分析的一个组成部分,我们还得到了新的添加剂和乘法形式的甲骨文风险不等式的分裂交叉验证,是独立的利益。
英文摘要:
Recent empirical and theoretical analyses of several commonly used prediction procedures reveal a peculiar risk behavior in high dimensions, referred to as double/multiple descent, in which the asymptotic risk is a non-monotonic function of the limiting aspect ratio of the number of features or parameters to the sample size. To mitigate this undesirable behavior, we develop a general framework for risk monotonization based on cross-validation that takes as input a generic prediction procedure and returns a modified procedure whose out-of-sample prediction risk is, asymptotically, monotonic in the limiting aspect ratio. As part of our framework, we propose two data-driven methodologies, namely zero- and one-step, that are akin to bagging and boosting, respectively, and show that, under very mild assumptions, they provably achieve monotonic asymptotic risk behavior. Our results are applicable to a broad variety of prediction procedures and loss functions, and do not require a well-specified (parametric) model. We exemplify our framework with concrete analyses of the minimum $\ell_2$, $\ell_1$-norm least squares prediction procedures. As one of the ingredients in our analysis, we also derive novel additive and multiplicative forms of oracle risk inequalities for split cross-validation that are of independent interest.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。