Tight PAC-Bayesian Risk Certificates for Contrastive Learning
LTCI, Télécom Paris, Institut Polytechnique de Paris（LTCI，巴黎电信学院，巴黎高等理工学院）；School of Computation Information and Technology, Technical University of Munich（计算信息与技术学院，慕尼黑技术大学）
Anna Van Elst, Debarghya Ghoshdastidar
发表信息：
SIAM Journal on Mathematics of Data Science, vol. 7, no. 4, pp. 1904-1927, 2025
Contrastive representation learning is a modern paradigm for learning representations of unlabeled data via augmentations -- precisely, contrastive models learn to embed semantically similar pairs of samples (positive pairs) closer than independently drawn samples (negative samples). In spite of its empirical success and widespread use in foundation models, statistical theory for contrastive learning remains less explored. Recent works have developed generalization error bounds for contrastive losses, but the resulting risk certificates are either vacuous (certificates based on Rademacher complexity or $f$-divergence) or require strong assumptions about samples that are unreasonable in practice. The present paper develops non-vacuous PAC-Bayesian risk certificates for contrastive representation learning, considering the practical considerations of the popular SimCLR framework. Notably, we take into account that SimCLR reuses positive pairs of augmented data as negative samples for other data, thereby inducing strong dependence and making classical PAC or PAC-Bayesian bounds inapplicable. We further refine existing bounds on the downstream classification loss by incorporating SimCLR-specific factors, including data augmentation and temperature scaling, and derive risk certificates for the contrastive zero-one risk. The resulting bounds for contrastive loss and downstream prediction are much tighter than those of previous risk certificates, as demonstrated by experiments on CIFAR-10.
A PAC-Bayesian View of Generalisation for Physics-Informed Machine Learning
Université Jean Monnet Saint-Étienne, CNRS, Institut d’Optique Graduate School, Laboratoire Hubert Curien UMR 5516（里昂蒙特大学圣埃蒂安分校、法国国家科学研究中心、光学研究生院、Hubert Curien实验室 UMR 5516）；Inria and University College London, France and United Kingdom（Inria 和 英国伦敦大学学院，法国和英国）
The University of Tokyo（东京大学）；RIKEN（日本理化学研究所）；Université Laval（拉瓦尔大学）；Inria（法国国家信息与自动化研究所）；University College London（伦敦大学学院）
Contrastive unsupervised representation learning (CURL) is the state-of-the-art technique to learn representations (as a set of features) from unlabelled data. While CURL has collected several empirical successes recently, theoretical understanding of its performance was still missing. In a recent work, Arora et al. (2019) provide the first generalisation bounds for CURL, relying on a Rademacher complexity. We extend their framework to the flexible PAC-Bayes setting, allowing us to deal with the non-iid setting. We present PAC-Bayesian generalisation bounds for CURL, which are then used to der…
Contrastive Self-Supervised Learning As Neural Manifold Packing
Center for Soft Matter Research, Department of Physics, New York University（纽约大学软物质研究中心）；Center for Neural Science, New York University（纽约大学神经科学中心）；Department of Psychology, New York University（纽约大学心理学系）；Simons Center for Computational Physical Chemistry, New York University（纽约大学西蒙计算物理化学中心）；Courant Institute of Mathematical Sciences, New York University（纽约大学柯朗数学科学研究所）
One Train for Two Tasks: An Encrypted Traffic Classification Framework Using Supervised Contrastive Learning
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Nanjing University of Posts and Telecommunications（南京邮电大学）；Peng Cheng Laboratory（鹏城实验室）；Southeast University（东南大学）；National University of Singapore（新加坡国立大学）
PACD-Net: Pseudo-Augmented Contrastive Distillation for Glycemic Control Estimation from SMBG
University of Virginia, School of Data Science, Charlottesville, VA 22903, USA（弗吉尼亚大学数据科学学院）；University of Virginia, Department of Pediatrics, Charlottesville, VA 22903, USA（弗吉尼亚大学儿科系）
We present a new technique called contrastive principal component analysis (cPCA) that is designed to discover low-dimensional structure that is unique to a dataset, or enriched in one dataset relative to other data. The technique is a generalization of standard PCA, for the setting where multiple datasets are available -- e.g. a treatment and a control group, or a mixed versus a homogeneous population -- and the goal is to explore patterns that are specific to one of the datasets. We conduct a wide variety of experiments in which cPCA identifies important dataset-specific patterns that are mi…
Graduate School of Artificial Intelligence（人工智能研究生院）；POSTECH（浦项科技大学）
Conditional image generation is the task of generating diverse images using class label information. Although many conditional Generative Adversarial Networks (GAN) have shown realistic results, such methods consider pairwise relations between the embedding of an image and the embedding of the corresponding label (data-to-class relations) as the conditioning losses. In this paper, we propose ContraGAN that considers relations between multiple image embeddings in the same batch (data-to-data relations) as well as the data-to-class relations by using a conditional contrastive loss. The discrimin…
Self-Supervised Transformer-based Contrastive Learning for Intrusion Detection Systems
Dept. of Electrical and Computer Engineering, University of Thessaly（电气与计算机工程系，希腊塞萨洛尼基大学）；Centre for Research and Technology Hellas, CERTH（希腊研究中心与技术中心，CERTH）
University of Technology Sydney（悉尼科技大学）；Monash University（莫纳什大学）；Wuhan University of Technology（武汉理工大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
Unsupervised graph representation learning has emerged as a powerful tool to address real-world problems and achieves huge success in the graph learning domain. Graph contrastive learning is one of the unsupervised graph representation learning methods, which recently attracts attention from researchers and has achieved state-of-the-art performances on various tasks. The key to the success of graph contrastive learning is to construct proper contrasting pairs to acquire the underlying structural semantics of the graph. However, this key part is not fully explored currently, most of the ways ge…
Department of Clinical Biostatistics, Graduate School of Medical and Dental Sciences, Institute of Science Tokyo（东京科学大学医学齿科研究生院临床生物统计学系）；Biostatistics Division, Center for Research Administration and Support, National Cancer Center（国立癌症中心研究管理与支持中心生物统计学部）；Department Multidisciplinary Sciences, Graduate School of Arts and Sciences, The University of Tokyo（东京大学艺术与科学研究生院多学科科学系）；Department of Epidemiology, National Institute of Infectious Diseases, Japan Institute for Health Security（日本健康安全研究所国立传染病研究所流行病学系）
The performance of a reinforcement learning algorithm can vary drastically during learning because of exploration. Existing algorithms provide little information about the quality of their current policy before executing it, and thus have limited use in high-stakes applications like healthcare. We address this lack of accountability by proposing that algorithms output policy certificates. These certificates bound the sub-optimality and return of the policy in the next episode, allowing humans to intervene when the certified quality is not satisfactory. We further introduce two new algorithms w…
University College London（伦敦大学学院）；DeepMind Edmonton（深度思维埃德蒙顿）
This paper presents an empirical study regarding training probabilistic neural networks using training objectives derived from PAC-Bayes bounds. In the context of probabilistic neural networks, the output of training is a probability distribution over network weights. We present two training objectives, used here for the first time in connection with training neural networks. These two training objectives are derived from tight PAC-Bayes bounds. We also re-implement a previously used training objective based on a classical PAC-Bayes bound, to compare the properties of the predictors learned us…
MIETT: Multi-Instance Encrypted Traffic Transformer for Encrypted Traffic Classification
School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）
Fine-Grained Uncertainty Quantification via Collisions
Department of Electrical and Computer Engineering, The University of Arizona（电气与计算机工程系，亚利桑那大学）；Pacific Northwest National Laboratory（太平洋西北国家实验室）；Marvell Technology Inc.（Marvell技术公司）
Selective Risk Certification for LLM Outputs via Information-Lift Statistics: PAC-Bayes, Robustness, and Skeleton Design
Department of Computer Science Iowa State University（计算机科学系爱荷华州立大学）；Department of Civil, Construction and Environmental Engineering Iowa State University（土木、建设与环境工程系爱荷华州立大学）
We propose a novel class of deep stochastic predictors for classifying metric data on graphs within the PAC-Bayes risk certification paradigm. Classifiers are realized as linearly parametrized deep assignment flows with random initial conditions. Building on the recent PAC-Bayes literature and data-dependent priors, this approach enables (i) to use risk bounds as training objectives for learning posterior distributions on the hypothesis space and (ii) to compute tight out-of-sample risk certificates of randomized classifiers more efficiently than related work. Comparison with empirical test se…
