PAC-Bayesian Bounds for Randomized Empirical Risk Minimizers

PAC-Bayes 界友好入门

User-friendly introduction to PAC-Bayes bounds

本文为 PAC-Bayes 理论提供友好入门，介绍其核心概念、发展历程及近期应用，旨在填补该领域缺乏入门介绍的空白。

隐藏却可量化：利用随机试验确定混淆强度的下限

Hidden yet quantifiable: A lower bound for confounding strength using randomized trials

本文提出利用随机试验量化未观察到的混淆因素，设计统计检验检测强混淆并估计其下限，通过合成数据验证方法有效性，并在实际场景中识别混淆存在与否。

PAC-Bayes Analysis Beyond the Usual Bounds

We focus on a stochastic learning model where the learner observes a finite set of training examples and the output of the learning process is a data-dependent distribution over a space of hypotheses. The learned data-dependent distribution is then used to make randomized predictions, and the high-level theme addressed here is guaranteeing the quality of predictions on examples that were not seen during training, i.e. generalization. In this setting the unknown quantity of interest is the expected risk of the data-dependent randomized predictor, for which upper bounds can be derived via a PAC-…

异质性随机对照试验中用于至事件发生时间结局的亚组分析方法

Subgroup analysis methods for time-to-event outcomes in heterogeneous randomized controlled trials

本文针对至事件发生时间终点缺乏亚组分析系统评估的问题，提出合成与半合成数据生成流程并开源 Python 基准框架，围绕异质性存在性、相关生物标志物和良好应答者识别开展算法评测。

On Medians of (Randomized) Pairwise Means

Tournament procedures, recently introduced in Lugosi & Mendelson (2016), offer an appealing alternative, from a theoretical perspective at least, to the principle of Empirical Risk Minimization in machine learning. Statistical learning by Median-of-Means (MoM) basically consists in segmenting the training data into blocks of equal size and comparing the statistical performance of every pair of candidate decision rules on each data block: that with highest performance on the majority of the blocks is declared as the winner. In the context of nonparametric regression, functions having won all th…

Rademacher upper bounds for cross-validation errors with an application to the lasso

We establish a general upper bound for $K$-fold cross-validation ($K$-CV) errors that can be adapted to many $K$-CV-based estimators and learning algorithms. Based on Rademacher complexity of the model and the Orlicz-$Ψ_ν$ norm of the error process, the CV error upper bound applies to both light-tail and heavy-tail error distributions. We also extend the CV error upper bound to $β$-mixing data using the technique of independent blocking. We provide a Python package (\texttt{CVbound}, \url{https://github.com/isaac2math}) for computing the CV error upper bound in $K$-CV-based algorithms. Using t…

PAC-Bayes bounds for stable algorithms with instance-dependent priors

PAC-Bayes bounds have been proposed to get risk estimates based on a training sample. In this paper the PAC-Bayes approach is combined with stability of the hypothesis learned by a Hilbert space valued algorithm. The PAC-Bayes setting is used with a Gaussian prior centered at the expected output. Thus a novelty of our paper is using priors defined in terms of the data-generating distribution. Our main result estimates the risk of the randomized algorithm in terms of the hypothesis stability coefficients. We also provide a new bound for the SVM classifier, which is compared to other known bound…

PAC-Bayes Un-Expected Bernstein Inequality

We present a new PAC-Bayesian generalization bound. Standard bounds contain a $\sqrt{L_n \cdot \KL/n}$ complexity term which dominates unless $L_n$, the empirical error of the learning algorithm's randomized predictions, vanishes. We manage to replace $L_n$ by a term which vanishes in many more situations, essentially whenever the employed learning algorithm is sufficiently stable on the dataset at hand. Our new bound consistently beats state-of-the-art bounds both on a toy example and on UCI datasets (with large enough $n$). Theoretically, unlike existing bounds, our new bound can be expected…

Self-Certifying Classification by Linearized Deep Assignment

We propose a novel class of deep stochastic predictors for classifying metric data on graphs within the PAC-Bayes risk certification paradigm. Classifiers are realized as linearly parametrized deep assignment flows with random initial conditions. Building on the recent PAC-Bayes literature and data-dependent priors, this approach enables (i) to use risk bounds as training objectives for learning posterior distributions on the hypothesis space and (ii) to compute tight out-of-sample risk certificates of randomized classifiers more efficiently than related work. Comparison with empirical test se…

Entropy-SGD optimizes the prior of a PAC-Bayes bound: Generalization properties of Entropy-SGD and data-dependent priors

We show that Entropy-SGD (Chaudhari et al., 2017), when viewed as a learning algorithm, optimizes a PAC-Bayes bound on the risk of a Gibbs (posterior) classifier, i.e., a randomized classifier obtained by a risk-sensitive perturbation of the weights of a learned classifier. Entropy-SGD works by optimizing the bound's prior, violating the hypothesis of the PAC-Bayes theorem that the prior is chosen independently of the data. Indeed, available implementations of Entropy-SGD rapidly obtain zero training error on random labels and the same holds of the Gibbs posterior. In order to obtain a valid g…

是什么让基于森林的异质性处理效应估计量有效？

What Makes Forest-Based Heterogeneous Treatment Effect Estimators Work?

本文统一因果森林与基于模型的森林，在L2加性模型框架下剖析其计算要素，并通过随机与观察性基准发现处理指标的倾向评分局部中心化是抗混杂性能关键。