arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.13934cs.LGstat.ML
收藏

异质分布偏移下的统计学习

Statistical Learning under Heterogeneous Distribution Shift

MIT(麻省理工学院);MSR NY(微软研究院纽约分院)

Max Simchowitz, Anurag Ajay, Pulkit Agrawal, Akshay Krishnamurthy

AI总结:

本文研究异质协变量偏移下可加预测器的经验风险最小化性能,证明当函数类F比G更简单时,ERM恢复f分量的速率对G的复杂度仅低阶依赖,从而对x上的较大偏移更具鲁棒性,并通过实验验证了该结论。

摘要:

本文研究从一对随机变量 $(\mathbf{x},\mathbf{y})$ 预测目标 $\mathbf{z}$ 的问题,其中真实预测器具有可加结构 $\mathbb{E}[\mathbf{z} \mid \mathbf{x},\mathbf{y}] = f_\star(\mathbf{x}) + g_{\star}(\mathbf{y})$。我们研究经验风险最小化(ERM)在函数类 $f+g$、$f \in F$ 和 $g \in G$ 上的性能,其中模型在给定的训练分布上拟合,但在存在协变量偏移的测试分布上进行评估。我们证明,当函数类 $F$ 比 $G$ “更简单”(例如以其度量熵来衡量)时,我们的预测器对异质协变量偏移更具鲁棒性,即 $\mathbf{x}$ 上的偏移远大于 $\mathbf{y}$ 上的偏移。我们的分析通过证明 ERM 在定性上表现得与正交机器学习类似:ERM 恢复预测器中 $f$ 分量的速率对函数类 $G$ 的复杂度仅具有低阶依赖,并针对可加结构引入的部分不可识别性进行了调整。这些结果依赖于一个针对 Dudley 积分的新型 Hölder 型不等式,该不等式可能具有独立的研究价值。此外,我们通过实验验证了理论发现,证明在众多领域中,对“更简单”特征的偏移具有更强的鲁棒性。

英文摘要:

This paper studies the prediction of a target $\mathbf{z}$ from a pair of random variables $(\mathbf{x},\mathbf{y})$, where the ground-truth predictor is additive $\mathbb{E}[\mathbf{z} \mid \mathbf{x},\mathbf{y}] = f_\star(\mathbf{x}) +g_{\star}(\mathbf{y})$. We study the performance of empirical risk minimization (ERM) over functions $f+g$, $f \in F$ and $g \in G$, fit on a given training distribution, but evaluated on a test distribution which exhibits covariate shift. We show that, when the class $F$ is "simpler" than $G$ (measured, e.g., in terms of its metric entropy), our predictor is more resilient to heterogeneous covariate shifts} in which the shift in $\mathbf{x}$ is much greater than that in $\mathbf{y}$. Our analysis proceeds by demonstrating that ERM behaves qualitatively similarly to orthogonal machine learning: the rate at which ERM recovers the $f$-component of the predictor has only a lower-order dependence on the complexity of the class $G$, adjusted for partial non-indentifiability introduced by the additive structure. These results rely on a novel Hölder style inequality for the Dudley integral which may be of independent interest. Moreover, we corroborate our theoretical findings with experiments demonstrating improved resilience to shifts in "simpler" features across numerous domains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑