arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.18115cs.LGcs.DSmath.OC
收藏

群体分布鲁棒优化神经元的鲁棒学习

Robust Learning of a Group DRO Neuron

University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Microsoft Research, Cambridge(微软研究院,剑桥)

Guyang Cao, Shuyao Li, Sushrut Karmalkar, Jelena Diakonikolas

AI总结:

针对任意标签噪声和群体级分布偏移,提出一种计算高效的原对偶算法,学习一个单神经元,使其在最小化最坏情况群体加权损失时达到常数因子竞争比。

摘要:

我们研究在存在任意标签噪声和群体级分布偏移的情况下,对于一大类协变量分布,在标准平方损失下学习单个神经元的问题。我们的目标是识别一个由 $\mathbf{w}_*$ 参数化的“最佳拟合”神经元,该神经元在最具挑战性的群体重新加权下表现良好。具体来说,我们解决了一个群体分布鲁棒优化问题:给定对 $K$ 个不同分布 $\mathcal p_{[1]},\dots,\mathcal p_{[K]}$ 的样本访问,我们寻求近似 $\mathbf{w}_*$,该 $\mathbf{w}_*$ 最小化群体分布的凸组合 $\boldsymbolλ \in Δ_K$ 上的最坏情况目标,其中目标为 $\sum_{i \in [K]}λ_{[i]}\,\mathbb E_{(\mathbf x,y)\sim\mathcal p_{[i]}}(σ(\mathbf w\cdot\mathbf x)-y)^2 - νd_f(\boldsymbolλ,\frac{1}{K}\mathbf1)$,而 $d_f$ 是一个 $f$-散度,用于对偏离均匀群体权重施加(可选的)惩罚,由参数 $ν\geq 0$ 缩放。我们开发了一个计算高效的原对偶算法,输出一个向量 $\widehat{\mathbf w}$,该向量在最坏情况群体加权下与 $\mathbf{w}_*$ 相比具有常数因子竞争比。我们的分析框架直接应对损失函数固有的非凸性,在任意标签损坏和群体特定分布偏移的情况下提供鲁棒学习保证。受我们算法框架启发的对偶外推更新实现,在 LLM 预训练基准测试中显示出前景。

英文摘要:

We study the problem of learning a single neuron under standard squared loss in the presence of arbitrary label noise and group-level distributional shifts, for a broad family of covariate distributions. Our goal is to identify a ''best-fit'' neuron parameterized by $\mathbf{w}_*$ that performs well under the most challenging reweighting of the groups. Specifically, we address a Group Distributionally Robust Optimization problem: given sample access to $K$ distinct distributions $\mathcal p_{[1]},\dots,\mathcal p_{[K]}$, we seek to approximate $\mathbf{w}_*$ that minimizes the worst-case objective over convex combinations of group distributions $\boldsymbolλ \in Δ_K$, where the objective is $\sum_{i \in [K]}λ_{[i]}\,\mathbb E_{(\mathbf x,y)\sim\mathcal p_{[i]}}(σ(\mathbf w\cdot\mathbf x)-y)^2 - νd_f(\boldsymbolλ,\frac{1}{K}\mathbf1)$ and $d_f$ is an $f$-divergence that imposes (optional) penalty on deviations from uniform group weights, scaled by a parameter $ν\geq 0$. We develop a computationally efficient primal-dual algorithm that outputs a vector $\widehat{\mathbf w}$ that is constant-factor competitive with $\mathbf{w}_*$ under the worst-case group weighting. Our analytical framework directly confronts the inherent nonconvexity of the loss function, providing robust learning guarantees in the face of arbitrary label corruptions and group-specific distributional shifts. The implementation of the dual extrapolation update motivated by our algorithmic framework shows promise on LLM pre-training benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑