arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.01233math.STcs.LGstat.MLstat.TH
收藏

面向分布外预测的最优岭正则化

Optimal Ridge Regularization for Out-of-Distribution Prediction

University of California, Berkeley(加州大学伯克利分校);Carnegie Mellon University(卡内基梅隆大学)

Pratik Patil, Jin-Hong Du, Ryan J. Tibshirani

AI总结:

本文研究分布外预测中最优岭正则化的行为,给出决定最优正则化水平符号的一般条件,并证明最优调参风险关于数据纵横比单调,揭示了负正则化在协变量或回归偏移下可能最优的现象。

摘要:

我们研究了分布外预测中最优岭正则化与最优岭风险的行为,其中测试分布可任意偏离训练分布。我们建立了在协变量偏移和回归偏移下决定最优正则化水平符号的一般条件。这些条件刻画了训练数据与测试数据中协方差结构与信号结构之间的一致性,并揭示了与分布内设定相比的显著差异。例如,在协变量偏移或回归偏移下,即使训练特征是各向同性的或设计是欠参数化的,负正则化水平也可能是最优的。此外,我们证明了最优调参后的风险关于数据纵横比是单调的,即使在分布外设定以及在对负正则化水平进行优化时也是如此。总体而言,除矩有界外,我们的结果不对训练分布或测试分布作任何建模假设,并允许任意偏移以及尽可能宽的(负)正则化水平范围。

英文摘要:

We study the behavior of optimal ridge regularization and optimal ridge risk for out-of-distribution prediction, where the test distribution deviates arbitrarily from the train distribution. We establish general conditions that determine the sign of the optimal regularization level under covariate and regression shifts. These conditions capture the alignment between the covariance and signal structures in the train and test data and reveal stark differences compared to the in-distribution setting. For example, a negative regularization level can be optimal under covariate shift or regression shift, even when the training features are isotropic or the design is underparameterized. Furthermore, we prove that the optimally-tuned risk is monotonic in the data aspect ratio, even in the out-of-distribution setting and when optimizing over negative regularization levels. In general, our results do not make any modeling assumptions for the train or the test distributions, except for moment bounds, and allow for arbitrary shifts and the widest possible range of (negative) regularization levels.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑