通过转导先验得到无界损失下的精细化风险界
Refined Risk Bounds for Unbounded Losses via Transductive Priors
- Department of Electrical Engineering and Computer Science Massachusetts Institute of Technology（电气工程与计算机科学系 马萨诸塞理工学院）
- Department of Brain and Cognitive Sciences Massachusetts Institute of Technology（脑科学与认知科学系 马萨诸塞理工学院）
- Department of Statistics University of California, Berkeley（统计学系 加州大学伯克利分校）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
该研究在转导在线学习设置下，利用带转导先验的指数权重算法，推导了无界损失下更优的风险与遗憾界，部分界不依赖设计向量或最优解范数，且算法有多项式时间近似。
AI中文摘要：
我们重新研究了带平方损失的线性回归、带hinge损失的分类问题以及逻辑回归的序列变体，这些问题均以无界损失为特征，且我们不对设计向量的幅度和最优参数向量的范数做任何假设。与现有结果的关键区别在于，我们假设设计向量集合是预先已知的（尽管它们的顺序未知），这种设置有时被称为转导在线学习。虽然这一假设看起来与固定设计回归或去噪类似，但我们证明，我们算法的序列特性允许我们在不对设计向量的分布做任何额外假设的情况下，将我们的界转化为随机设计下的统计界——这对于标准去噪结果而言是不可能的。我们的核心工具基于带有精心选择的转导（依赖于设计的）先验的指数权重算法，该算法利用了设计向量的完整时间跨度。\n我们的分类遗憾界具有一个在现有文献中仅属于有界损失的特性：它们仅依赖于参数空间的维度和轮数，与设计向量或最优解的范数无关。对于带平方损失的线性回归，我们进一步将分析扩展到稀疏情形，给出了额外依赖于响应变量幅度的稀疏遗憾界。我们认为，这些改进的界是转导设置所特有的，在最坏情况序列设置中是无法达到的。在多种情况下，我们的算法具有多项式时间近似，并且简化为对对数凹测度进行采样，而非对难以构造的类的ε-覆盖进行聚合。
英文摘要：
We revisit the sequential variants of linear regression with the squared loss, classification problems with hinge loss, and logistic regression, all characterized by unbounded losses in the setup where no assumptions are made on the magnitude of design vectors and the norm of the optimal vector of parameters. The key distinction from existing results lies in our assumption that the set of design vectors is known in advance (though their order is not), a setup sometimes referred to as transductive online learning. While this assumption seems similar to fixed design regression or denoising, we demonstrate that the sequential nature of our algorithms allows us to convert our bounds into statistical ones with random design without making any additional assumptions about the distribution of the design vectors--an impossibility for standard denoising results. Our key tools are based on the exponential weights algorithm with carefully chosen transductive (design-dependent) priors, which exploit the full horizon of the design vectors. Our classification regret bounds have a feature that is only attributed to bounded losses in the literature: they depend solely on the dimension of the parameter space and on the number of rounds, independent of the design vectors or the norm of the optimal solution. For linear regression with squared loss, we further extend our analysis to the sparse case, providing sparsity regret bounds that additionally depend on the magnitude of the response variables. We argue that these improved bounds are specific to the transductive setting and unattainable in the worst-case sequential setup. Our algorithms, in several cases, have polynomial time approximations and reduce to sampling with respect to log-concave measures instead of aggregating over hard-to-construct $\varepsilon$-covers of classes.
