arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.20967stat.MLcs.LGmath.STstat.TH
收藏

从稀疏目标采样的硬标签误导旋转不变算法

Hard labels sampled from sparse targets mislead rotation invariant algorithms

University of California, Berkeley(加州大学伯克利分校);University of Wisconsin, Madison(威斯康星大学麦迪逊分校)

Avrajit Ghosh, Bin Yu, Manfred Warmuth, Peter Bartlett

AI总结:

针对稀疏目标下的二分类问题,证明旋转不变算法(如逻辑损失梯度下降)的过风险下界为Ω((d-1)/n),而通过重参数化u_i v_i的非旋转不变算法可实现O(s log d / n)的上界。

摘要:

最常见的机器学习设置之一是逻辑回归。在许多分类模型中,包括神经网络,最终预测是通过将逻辑链接函数应用于线性得分获得的。在二元逻辑回归中,反馈可以是软标签(对应于数据的真实条件概率,如在蒸馏中)或采样的硬标签(取值为$\pm 1$)。我们指出即使在特别有利的设置中也会出现一个基本问题,其中目标是学习形式为$\sigma(\mathbf{x}^{\top}\mathbf{w}^{\star})$的无噪声软目标。在过约束情况(即样本数$n$超过输入维度$d$)下,使用样本$(\mathbf{x}_i,\sigma(\mathbf{x}_i^{\top}\mathbf{w}^{\star}))$足以恢复$\mathbf{w}^{\star}$,从而获得贝叶斯风险。然而,我们证明当样本由从相同条件分布$\sigma(\mathbf{x}_i^{\top}\mathbf{w}^{\star})$采样的硬标签$y_i$标记,且$\mathbf{w}^{\star}$是$s$-稀疏时,旋转不变算法被证明是次优的:它们产生过风险$\Omega\\!\left(\frac{d-1}{n}\right)$,而存在简单的非旋转不变算法,其过风险为$O(\frac{s\log d}{n})$。最简单的旋转不变算法是逻辑损失上的梯度下降(带早停)。针对稀疏目标实现上述上界的简单非旋转不变算法使用对权重$u_i,v_i$的梯度下降,其中线性权重$w_i$被重参数化为$u_i v_i$。

英文摘要:

One of the most common machine learning setups is logistic regression. In many classification models, including neural networks, the final prediction is obtained by applying a logistic link function to a linear score. In binary logistic regression, the feedback can be either soft labels, corresponding to the true conditional probability of the data (as in distillation), or sampled hard labels (taking values $\pm 1$). We point out a fundamental problem that arises even in a particularly favorable setting, where the goal is to learn a noise-free soft target of the form $σ(\mathbf{x}^{\top}\mathbf{w}^{\star})$. In the over-constrained case (i.e. the number of samples $n$ exceeds the input dimension $d$) with examples $(\mathbf{x}_i,σ(\mathbf{x}_i^{\top}\mathbf{w}^{\star}))$, it is sufficient to recover $\mathbf{w}^{\star}$ and hence achieve the Bayes risk. However, we prove that when the examples are labeled by hard labels $y_i$ sampled from the same conditional distribution $σ(\mathbf{x}_i^{\top}\mathbf{w}^{\star})$ and $\mathbf{w}^{\star}$ is $s$-sparse, then rotation-invariant algorithms are provably suboptimal: they incur an excess risk $Ω\!\left(\frac{d-1}{n}\right)$, while there are simple non-rotation invariant algorithms with excess risk $O(\frac{s\log d}{n})$. The simplest rotation invariant algorithm is gradient descent on the logistic loss (with early stopping). A simple non-rotation-invariant algorithm for sparse targets that achieves the above upper bounds uses gradient descent on the weights $u_i,v_i$, where now the linear weight $w_i$ is reparameterized as $u_iv_i$.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑