非各向同性随机张量的集中性及其在学习和经验风险最小化中的应用

Learning without Concentration for General Loss Functions

We study prediction and estimation problems using empirical risk minimization, relative to a general convex loss function. We obtain sharp error rates even when concentration is false or is very restricted, for example, in heavy-tailed scenarios. Our results show that the error rate depends on two parameters: one captures the intrinsic complexity of the class, and essentially leads to the error rate in a noise-free (or realizable) problem; the other measures interactions between class members the target and the loss, and is dominant when the problem is far from realizable. We also explain how …

从非独立同分布且非各向同性数据中学习样本高效的线性表示

Sample-Efficient Linear Representation Learning from Non-IID Non-Isotropic Data

针对非独立同分布且非各向同性数据，提出DFW算法改进交替最小化下降，消除表示学习偏差，实现随总数据量降低的噪声缩放和线性收敛，提升了样本效率与泛化能力。

方差最优臂选择：误分配最小化与最佳臂识别

Variance-Optimal Arm Selection: Misallocation Minimization and Best Arm Identification

本文提出UCB-VV和SHVV算法，分别解决误分配最小化和固定预算最佳臂识别问题，通过理论分析和实验证明其在方差最优选择中的有效性。

2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization

Let $\mathcal{F}$ be a class of measurable functions $f:S\mapsto [0,1]$ defined on a probability space $(S,\mathcal{A},P)$. Given a sample (X_1,...,X_n) of i.i.d. random variables taking values in S with common distribution P, let P_n denote the empirical measure based on (X_1,...,X_n). We study an empirical risk minimization problem $P_nf\to \min$, $f\in \mathcal{F}$. Given a solution $\hat{f}_n$ of this problem, the goal is to obtain very general upper bounds on its excess risk \[\mathcal{E}_P(\hat{f}_n):=P\hat{f}_n-\inf_{f\in \mathcal{F}}Pf,\] expressed in terms of relevant geometric parame…

高维经验风险最小化中高斯普适性破坏的表征

Characterization of Gaussian Universality Breakdown in High-Dimensional Empirical Risk Minimization

通过将凸高斯极小极大定理推广到非高斯数据，刻画了高维经验风险最小化估计量的渐近分布，揭示了高斯普适性的适用范围与局限。

From inexact optimization to learning via gradient concentration

Optimization in machine learning typically deals with the minimization of empirical objectives defined by training data. However, the ultimate goal of learning is to minimize the error on future data (test error), for which the training data provides only partial information. In this view, the optimization problems that are practically feasible are based on inexact quantities that are stochastic in nature. In this paper, we show how probabilistic results, specifically gradient concentration, can be combined with results from inexact optimization to derive sharp test error guarantees. By consid…

Conditional Risk Minimization for Stochastic Processes

We study the task of learning from non-i.i.d. data. In particular, we aim at learning predictors that minimize the conditional risk for a stochastic process, i.e. the expected loss of the predictor on the next point conditioned on the set of training samples observed so far. For non-i.i.d. data, the training set contains information about the upcoming samples, so learning with respect to the conditional distribution can be expected to yield better predictors than one obtains from the classical setting of minimizing the marginal risk. Our main contribution is a practical estimator for the condi…

通过核范数最小化进行张量补全

On Tensor Completion via Nuclear Norm Minimization

本文提出通过直接最小化张量核范数进行张量补全，证明该方法能提升采样需求，发展了张量核范数的子微分刻画和张量鞅集中不等式等技术。

Learning Parameters for Weighted Matrix Completion via Empirical Estimation

Recently theoretical guarantees have been obtained for matrix completion in the non-uniform sampling regime. In particular, if the sampling distribution aligns with the underlying matrix's leverage scores, then with high probability nuclear norm minimization will exactly recover the low rank matrix. In this article, we analyze the scenario in which the non-uniform sampling distribution may or may not not align with the underlying matrix's leverage scores. Here we explore learning the parameters for weighted nuclear norm minimization in terms of the empirical sampling distribution. We provide a…