首页/论文/2207.02440

arXiv 2207.024402022-07cs.LGstat.ML

PAC Prediction Sets for Meta-Learning

Georgia Institute of Technology（佐治亚理工学院）；University of Pennsylvania（宾夕法尼亚大学）

Sangdon Park, Edgar Dobriban, Insup Lee, Osbert Bastani

备注：: Accepted to NeurIPS 2022

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

Uncertainty quantification is a key component of machine learning models targeted at safety-critical systems such as in healthcare or autonomous vehicles. We study this problem in the context of meta learning, where the goal is to quickly adapt a predictor to new tasks. In particular, we propose a novel algorithm to construct \emph{PAC prediction sets}, which capture uncertainty via sets of labels, that can be adapted to new tasks with only a few training examples. These prediction sets satisfy an extension of the typical PAC guarantee to the meta learning setting; in particular, the PAC guarantee holds with high probability over future tasks. We demonstrate the efficacy of our approach on four datasets across three application domains: mini-ImageNet and CIFAR10-C in the visual domain, FewRel in the language domain, and the CDC Heart Dataset in the medical domain. In particular, our prediction sets satisfy the PAC guarantee while having smaller size compared to other baselines that also satisfy this guarantee.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

PACOH: Bayes-Optimal Meta-Learning with PAC-Guarantees

ETH Zurich（苏黎世联邦理工学院）；EPFL（洛桑联邦理工学院）

Meta-learning can successfully acquire useful inductive biases from data. Yet, its generalization properties to unseen learning tasks are poorly understood. Particularly if the number of meta-training tasks is small, this raises concerns about overfitting. We provide a theoretical analysis using the PAC-Bayesian framework and derive novel generalization bounds for meta-learning. Using these bounds, we develop a class of PAC-optimal meta-learning algorithms with performance guarantees and a principled meta-level regularization. Unlike previous PAC-Bayesian meta-learners, our method results in a…

2021-06-18cs.LGstat.ML

Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learning

Google Research（谷歌研究院）

Despite recent advances in its theoretical understanding, there still remains a significant gap in the ability of existing PAC-Bayesian theories on meta-learning to explain performance improvements in the few-shot learning setting, where the number of training examples in the target tasks is severely limited. This gap originates from an assumption in the existing theories which supposes that the number of training examples in the observed tasks and the number of training examples in the target tasks follow the same distribution, an assumption that rarely holds in practice. By relaxing this ass…

2021-10-25cs.LGstat.ML

可扩展的PAC-贝叶斯元学习：从理论到实践的PAC最优超后验

Scalable PAC-Bayesian Meta-Learning via the PAC-Optimal Hyper-Posterior: From Theory to Practice

ETH Zurich（苏黎世联邦理工学院）；EPFL Lausanne（洛桑联邦理工学院）；Helmholtz AI（亥姆霍兹人工智能）；TU Munich（慕尼黑工业大学）

本文通过PAC-贝叶斯理论推导出最优超后验PACOH的闭式解，提出可扩展的元学习方法，避免双层优化，实验表明其在预测精度和不确定性估计上达到最先进水平。

2023-12-22cs.LGstat.ML

PAC-Bayes meta-learning with implicit task-specific posteriors

Australian Institute for Machine Learning, University of Adelaide（阿德莱德大学机器学习研究所）；Monash University（莫纳什大学）

We introduce a new and rigorously-formulated PAC-Bayes meta-learning algorithm that solves few-shot learning. Our proposed method extends the PAC-Bayes framework from a single task setting to the meta-learning multiple task setting to upper-bound the error evaluated on any, even unseen, tasks and samples. We also propose a generative-based approach to estimate the posterior of task-specific model parameters more expressively compared to the usual assumption based on a multivariate normal distribution with a diagonal covariance matrix. We show that the models trained with our proposed meta-lear…

2021-10-30cs.LGstat.ML

通过学习学习算法实现更灵活的PAC-Bayesian元学习

More Flexible PAC-Bayesian Meta-Learning by Learning Learning Algorithms

Institute of Science and Technology Austria (ISTA)（奥地利科学技术研究所）；Sharif University of Technology（谢里夫理工大学）

我们引入了一个基于PAC-Bayesian理论的元学习新框架，通过将元学习直接表达为学习未来任务的学习算法，实现了更灵活的任务间知识转移，并在理论和实验上均证明了其能提升预测质量。

2024-05-29cs.LGstat.ML

Far-HO: A Bilevel Programming Package for Hyperparameter Optimization and Meta-Learning

Istituto Italiano di Tecnologia（意大利理工学院）；University College London（伦敦大学学院）；Università degli studi di Firenze（佛罗伦萨大学）

In (Franceschi et al., 2018) we proposed a unified mathematical framework, grounded on bilevel programming, that encompasses gradient-based hyperparameter optimization and meta-learning. We formulated an approximate version of the problem where the inner objective is solved iteratively, and gave sufficient conditions ensuring convergence to the exact problem. In this work we show how to optimize learning rates, automatically weight the loss of single examples and learn hyper-representations with Far-HO, a software package based on the popular deep learning framework TensorFlow that allows to s…

2018-06-13cs.LGcs.MSstat.ML

更多相关论文收起更多论文

线性动力系统少样本识别的PAC-Bayesian元学习

PAC-Bayesian Meta-Learning for Few-Shot Identification of Linear Dynamical Systems

University of California, Los Angeles（加利福尼亚大学洛杉矶分校）

针对短轨迹、噪声大的LTI系统识别，提出PBML-LTI元学习框架，学习可迁移先验并进行贝叶斯自适应，通过鞅PAC-Bayes分析提供有限样本保证。

2026-09-22cs.LG

PAC Confidence Sets for Deep Neural Networks via Calibrated Prediction

University of Pennsylvania（宾夕法尼亚大学）

We propose an algorithm combining calibrated prediction and generalization bounds from learning theory to construct confidence sets for deep neural networks with PAC guarantees---i.e., the confidence set for a given input contains the true label with high probability. We demonstrate how our approach can be used to construct PAC confidence sets on ResNet for ImageNet, a visual object tracking model, and a dynamics model for the half-cheetah reinforcement learning problem.

2020-02-15cs.LGstat.ML

PAC-Bayes Bounds for Meta-learning with Data-Dependent Prior

By leveraging experience from previous tasks, meta-learning algorithms can achieve effective fast adaptation ability when encountering new tasks. However it is unclear how the generalization property applies to new tasks. Probably approximately correct (PAC) Bayes bound theory provides a theoretical framework to analyze the generalization performance for meta-learning. We derive three novel generalisation error bounds for meta-learning based on PAC-Bayes relative entropy bound. Furthermore, using the empirical risk minimization (ERM) method, a PAC-Bayes bound for meta-learning with data-depend…

2021-02-07cs.LGstat.ML

基于随机集PAC-Bayesian理论的数据依赖假设集的统一泛化界

Uniform Generalization Bounds on Data-Dependent Hypothesis Sets via PAC-Bayesian Theory on Random Sets

INRIA（法国国家信息与自动化研究所）；PSL Research University（巴黎文理研究大学）；Univ Rennes（雷恩大学）；CNRS（法国国家科学研究中心）；University of Oxford（牛津大学）

本文通过将PAC-Bayesian理论严格应用于随机集，提出数据依赖的统一泛化界，并用于统一基于分形维数的泛化界及证明Langevin动力学轨迹上的新泛化性质。

2025-02-10cs.LGstat.ML

A General framework for PAC-Bayes Bounds for Meta-Learning

Chalmers University of Technology（查尔姆斯理工大学）

Meta learning automatically infers an inductive bias, that includes the hyperparameter of the base-learning algorithm, by observing data from a finite number of related tasks. This paper studies PAC-Bayes bounds on meta generalization gap. The meta-generalization gap comprises two sources of generalization gaps: the environment-level and task-level gaps resulting from observation of a finite number of tasks and data samples per task, respectively. In this paper, by upper bounding arbitrary convex functions, which link the expected and empirical losses at the environment and also per-task level…

2022-06-11cs.LGstat.ML

标签偏移下的 PAC 预测集

PAC Prediction Sets Under Label Shift

University of Pennsylvania（宾夕法尼亚大学）；POSTECH（浦项科技大学）

针对分布偏移下保形预测集失效问题，提出一种在标签偏移下通过估计目标域类别概率和混淆矩阵并利用高斯消元算法计算重要性权重置信区间来构建预测集的新算法，在五个数据集上验证其满足PAC保证且预测集更小更具信息量。

2023-10-19cs.LGstat.ML

PAC Prediction Sets Under Covariate Shift

University of Pennsylvania（宾夕法尼亚大学）

An important challenge facing modern machine learning is how to rigorously quantify the uncertainty of model predictions. Conveying uncertainty is especially important when there are changes to the underlying data distribution that might invalidate the predictive model. Yet, most existing uncertainty quantification algorithms break down in the presence of such shifts. We propose a novel approach that addresses this challenge by constructing \emph{probably approximately correct (PAC)} prediction sets in the presence of covariate shift. Our approach focuses on the setting where there is a covari…

2022-03-16cs.LGstat.ML

Generalization Bounds for Meta-Learning via PAC-Bayes and Uniform Stability

Princeton University（普林斯顿大学）

We are motivated by the problem of providing strong generalization guarantees in the context of meta-learning. Existing generalization bounds are either challenging to evaluate or provide vacuous guarantees in even relatively simple settings. We derive a probably approximately correct (PAC) bound for gradient-based meta-learning using two different generalization frameworks in order to deal with the qualitatively different challenges of generalization at the "base" and "meta" levels. We employ bounds for uniformly stable algorithms at the base level and bounds from the PAC-Bayes framework at t…

2021-10-26cs.AIcs.LGstat.ML

代码大型语言模型的PAC预测集

PAC Prediction Sets for Large Language Models of Code

University of Pennsylvania（宾夕法尼亚大学）

本文提出利用抽象语法树生成紧凑的PAC预测集，以部分程序形式表示代码生成模型的不确定性，并在PICARD和Codex上验证了有效性，为首个此类研究。

2023-06-21cs.LG

Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory

Technion - Israel Institute of Technology（以色列理工学院）

In meta-learning an agent extracts knowledge from observed tasks, aiming to facilitate learning of novel future tasks. Under the assumption that future tasks are 'related' to previous tasks, the accumulated knowledge should be learned in a way which captures the common structure across learned tasks, while allowing the learner sufficient flexibility to adapt to novel aspects of new tasks. We present a framework for meta-learning that is based on generalization error bounds, allowing us to extend various PAC-Bayes bounds to meta-learning. Learning takes place through the construction of a distr…

2019-05-20cs.AIcs.LGstat.ML

基于元学习模型表示的泛化界：PAC-Bayes与样本压缩超网络

Generalization Bounds via Meta-Learned Model Representations: PAC-Bayes and Sample Compression Hypernetworks

ServiceNow Research（ServiceNow研究院）

本文提出一种元学习方案，利用超网络从数据集生成下游预测器参数，并通过PAC-Bayes编码器、样本压缩编码器及其混合架构推导紧致泛化界。

2025-06-05cs.LG