arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.14645cs.LGcs.CCcs.DS
收藏

回归场景下的全预测器与凸函数的近似秩

Omnipredictors for Regression and the Approximate Rank of Convex Functions

Apple(苹果公司);Cornell University(康奈尔大学);UC Berkeley(加州大学伯克利分校)

Parikshit Gopalan, Princewill Okoroafor, Prasad Raghavendra, Abhishek Shetty, Mihir Singhal

AI总结:

该研究将全预测器概念拓展至回归场景,提出损失最小化充分统计量概念,证明区间上凸Lipschitz函数的ε-近似秩上界并验证其紧性,还给出多类损失下的高效全预测器构造方法。

摘要:

考虑监督学习场景,其目标是学习从某分布中采样的点$\boldsymbol{x}$来预测标签$\boldsymbol{y}$。针对损失函数类$\boldsymbol{\text{Omnipredictor}}$(全预测器)和假设类$\boldsymbol{\text{Omnipredictor}}$的全预测器(omnipredictor)是这样一种预测器:对于$\boldsymbol{\text{Omnipredictor}}$中的每一个损失函数,其预测产生的期望损失都低于$\boldsymbol{\text{Omnipredictor}}$中最优假设的期望损失。自[GKR+21]提出该概念以来,已有大量研究围绕标签$\text{y} \text{in} \textbraceleft 0, 1 \textbraceright$的二分类场景展开,但针对标签$\text{y} \text{in} [0,1]$可为连续值的回归场景,相关研究成果仍较为有限。我们的核心概念贡献是提出了面向损失函数族损失最小化的充分统计量(sufficient statistics)概念:这类统计量是关于某分布的一组统计信息,掌握这些信息后,人们就能采取行动,使该损失族中任意损失函数的期望损失最小化。这一充分统计量概念与损失函数族的近似秩直接相关。我们的关键技术贡献是,给出了区间$[0,1]$上凸Lipschitz函数的$\boldsymbol{\text{ε}}$-近似秩的$O(1/\boldsymbol{\text{ε}}^{2/3})$上界,且我们证明该上界在相差一个$\text{polylog}(1/\boldsymbol{\text{ε}})$因子的意义下是紧的。在假设类$\boldsymbol{\text{Omnipredictor}}$满足弱可学习性的前提下,该结果提升了针对所有凸Lipschitz损失函数类的全预测器的学习运行效率。我们还针对损失族具有低次多项式近似、或源自广义线性模型(GLMs)的场景,给出了高效全预测器构造方法。我们将[GKH+23]为布尔标签场景提出的损失结果不可区分性技术推广至回归场景,才得以实现从充分统计量到更快速全预测器的转换。

英文摘要:

Consider the supervised learning setting where the goal is to learn to predict labels $\mathbf y$ given points $\mathbf x$ from a distribution. An \textit{omnipredictor} for a class $\mathcal L$ of loss functions and a class $\mathcal C$ of hypotheses is a predictor whose predictions incur less expected loss than the best hypothesis in $\mathcal C$ for every loss in $\mathcal L$. Since the work of [GKR+21] that introduced the notion, there has been a large body of work in the setting of binary labels where $\mathbf y \in \{0, 1\}$, but much less is known about the regression setting where $\mathbf y \in [0,1]$ can be continuous. Our main conceptual contribution is the notion of \textit{sufficient statistics} for loss minimization over a family of loss functions: these are a set of statistics about a distribution such that knowing them allows one to take actions that minimize the expected loss for any loss in the family. The notion of sufficient statistics relates directly to the approximate rank of the family of loss functions. Our key technical contribution is a bound of $O(1/\varepsilon^{2/3})$ on the $ε$-approximate rank of convex, Lipschitz functions on the interval $[0,1]$, which we show is tight up to a factor of $\mathrm{polylog} (1/ε)$. This yields improved runtimes for learning omnipredictors for the class of all convex, Lipschitz loss functions under weak learnability assumptions about the class $\mathcal C$. We also give efficient omnipredictors when the loss families have low-degree polynomial approximations, or arise from generalized linear models (GLMs). This translation from sufficient statistics to faster omnipredictors is made possible by lifting the technique of loss outcome indistinguishability introduced by [GKH+23] for Boolean labels to the regression setting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑