Arnak Dalalyan, Alexandre Tsybakov
We study the problem of aggregation under the squared loss in the model of regression with deterministic design. We obtain sharp PAC-Bayesian risk bounds for aggregates defined via exponential weights, under general assumptions on the distribution of errors and on the functions to aggregate. We then apply these results to derive sparsity oracle inequalities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
继续探索
The present paper is about estimation and prediction in high-dimensional additive models under a sparsity assumption ($p\gg n$ paradigm). A PAC-Bayesian strategy is investigated, delivering oracle inequalities in probability. The implementation is performed through recent outcomes in high-dimensional MCMC algorithms, and the performance of our method is assessed on simulated data.
Consider a regression model with fixed design and Gaussian noise where the regression function can potentially be well approximated by a function that admits a sparse representation in a given dictionary. This paper resorts to exponential weights to exploit this underlying sparsity by implementing the principle of sparsity pattern aggregation. This model selection take on sparse estimation allows us to derive sparsity oracle inequalities in several popular frameworks, including ordinary sparsity, fused sparsity and group sparsity. One striking aspect of these theoretical results is that they h…
We present a novel analysis of the expected risk of weighted majority vote in multiclass classification. The analysis takes correlation of predictions by ensemble members into account and provides a bound that is amenable to efficient minimization, which yields improved weighting for the majority vote. We also provide a specialized version of our bound for binary classification, which allows to exploit additional unlabeled data for tighter risk estimation. In experiments, we apply the bound to improve weighting of trees in random forests and show that, in contrast to the commonly used first or…
This paper is devoted to the bipartite ranking problem, a classical statistical learning task, in a high dimensional setting. We propose a scoring and ranking strategy based on the PAC-Bayesian approach. We consider nonlinear additive scoring functions, and we derive non-asymptotic risk bounds under a sparsity assumption. In particular, oracle inequalities in probability holding under a margin condition assess the performance of our procedure, and prove its minimax optimality. An MCMC-flavored algorithm is proposed to implement our method, along with its behavior on synthetic and real-life dat…
We consider a seller offering a large network of $N$ products over a time horizon of $T$ periods. The seller does not know the parameters of the products' linear demand model, and can dynamically adjust product prices to learn the demand model based on sales observations. The seller aims to minimize its pseudo-regret, i.e., the expected revenue loss relative to a clairvoyant who knows the underlying demand model. We consider a sparse set of demand relationships between products to characterize various connectivity properties of the product network. In particular, we study three different spars…
A sparse PAC-Bayesian approach for high-dimensional quantile prediction
本文提出一种面向高维分位数预测的稀疏PAC-Bayes方法，结合尺度化Student-t先验与Langevin Monte Carlo，通过非渐近oracle不等式证明其极小极大最优性与稀疏适应性，并在模拟和真实数据中验证了竞争力。
We combine Bayesian prediction and weighted inference as a unified approach to survey inference. The general principles of Bayesian analysis imply that models for survey outcomes should be conditional on all variables that affect the probability of inclusion. We incorporate the weighting variables under the framework of multilevel regression and poststratification, as a byproduct generating model-based weights after smoothing. We investigate deep interactions and introduce structured prior distributions for smoothing and stability of estimates. The computation is done via Stan and implemented …
Bayesian High-dimensional Linear Regression with Sparse Projection-posterior
提出一种基于稀疏投影后验的贝叶斯高维线性回归方法，实现估计、不确定性量化与变量选择，理论保证最优收缩率，并支持分布式计算，在模拟和真实数据中表现优异。
Assigning weights to a large pool of objects is a fundamental task in a wide variety of applications. In this article, we introduce the concept of structured high-dimensional probability simplexes, in which most components are zero or near zero and the remaining ones are close to each other. Such structure is well motivated by (i) high-dimensional weights that are common in modern applications, and (ii) ubiquitous examples in which equal weights -- despite their simplicity -- often achieve favorable or even state-of-the-art predictive performance. This particular structure, however, presents u…
PAC-Bayesian risk bounds for fully connected deep neural network with Gaussian priors
本文利用 PAC-Bayes 界研究高斯先验下全连接贝叶斯深度神经网络，建立了其在 Besov 空间中匹配极小极大最优速率的预测风险上界，适用于非参数回归和 logistic 损失二分类，并覆盖 Lipschitz 连续激活函数。
The popular Bayesian meta-analysis expressed by Bayesian normal-normal hierarchical model (NNHM) synthesizes knowledge from several studies and is highly relevant in practice. Moreover, NNHM is the simplest Bayesian hierarchical model (BHM), which illustrates problems typical in more complex BHMs. Until now, it has been unclear to what extent the data determines the marginal posterior distributions of the parameters in NNHM. To address this issue we computed the second derivative of the Bhattacharyya coefficient with respect to the weighted likelihood, defined the total empirical determinacy (…
Robust high-dimensional Bayesian regression with non-Gaussian errors under global--local shrinkage priors
针对高维多元回归中非高斯误差问题，提出基于尺度-位置混合误差分布与马蹄+先验的稳健贝叶斯框架，实现系数和残差精度矩阵的联合稀疏性，理论证明后验收缩性和选择一致性，模拟和实证表明优于高斯方法。
Leveraging Sparsity to Improve No-U-Turn Sampling Efficiency for Hierarchical Bayesian Models
SNUTS通过稀疏精度矩阵提升分层贝叶斯模型的采样效率，适用于高维、稀疏和高相关性场景。
We consider the problem of online linear regression on arbitrary deterministic sequences when the ambient dimension d can be much larger than the number of time rounds T. We introduce the notion of sparsity regret bound, which is a deterministic online counterpart of recent risk bounds derived in the stochastic setting under a sparsity scenario. We prove such regret bounds for an online-learning algorithm called SeqSEW and based on exponential weighting and data-driven truncation. In a second part we apply a parameter-free version of this algorithm to the stochastic setting (regression model w…
Controlling Multiple Errors Simultaneously with a PAC-Bayes Bound
针对现有PAC-Bayes泛化界仅支持标量性能指标的局限，提出首个约束M种误差类型的经验与真实概率KL散度的PAC-Bayes界，转化为可微训练目标，可同时控制所有误差加权组合，适配误分类严重程度动态变化的场景。
Debiased Inverse Propensity Score Weighting for Estimation of Average Treatment Effects with High-Dimensional Confounders
针对高维混杂下平均处理效应估计问题，提出去偏差逆倾向得分加权法，在倾向得分稀疏逻辑回归假设下获$\bigstar$-相合估计，支持复杂结果回归，可构建置信区间并达半参数效率，已发布R包dipw。
We consider the sparse regression model where the number of parameters $p$ is larger than the sample size $n$. The difficulty when considering high-dimensional problems is to propose estimators achieving a good compromise between statistical and computational performances. The BIC estimator for instance performs well from the statistical point of view \cite{BTW07} but can only be computed for values of $p$ of at most a few tens. The Lasso estimator is solution of a convex minimization problem, hence computable for large value of $p$. However stringent conditions on the design are required to e…
PAC-Bayes Bounds for High-Dimensional Multi-Index Models with Unknown Active Dimension
本文提出一种PAC-Bayesian估计方法，用于处理可能误设定的高维多索引模型，其中活跃维度未知。方法通过非渐近 oracle 不等式适应模型活跃维度、维度约简矩阵的稀疏性和链接函数的正则性。
Inferring Change Points in Regression via Sample Weighting
本文提出基于样本加权经验风险最小化的方法，用于高维广义线性模型中变化点的识别，通过样本权重编码先验信息，实现对M估计量和最大似然估计量的加权版本，并在高维极限下证明其性能。
扫码添加微信好友，提出您的宝贵建议 👇
💡 备注请填写：网站反馈
发表机构由 AI 从论文文本抽取，如发现错误，欢迎反馈修正建议。