Recent literature in self-supervised has demonstrated significant progress in closing the gap between supervised and unsupervised methods in the image and text domains. These methods rely on domain-specific augmentations that are not directly amenable to the tabular domain. Instead, we introduce Contrastive Mixup, a semi-supervised learning framework for tabular data and demonstrate its effectiveness in limited annotated data settings. Our proposed method leverages Mixup-based augmentation under the manifold assumption by mapping samples to a low dimensional latent space and encourage interpolated samples to have high a similarity within the same labeled class. Unlabeled samples are additionally employed via a transductive label propagation method to further enrich the set of similar and dissimilar pairs that can be used in the contrastive loss term. We demonstrate the effectiveness of the proposed framework on public tabular datasets and real-world clinical datasets.
This paper proposes a novel formulation of prototypical loss with mixup for speaker verification. Mixup is a simple yet efficient data augmentation technique that fabricates a weighted combination of random data point and label pairs for deep neural network training. Mixup has attracted increasing attention due to its ability to improve robustness and generalization of deep neural networks. Although mixup has shown success in diverse domains, most applications have centered around closed-set classification tasks. In this work, we propose contrastive-mixup, a novel augmentation strategy that le…
Contrastive Domain Adaptation for Time-Series via Temporal Mixup
Nanyang Technological University（南洋理工大学）；Agency for Science, Technology and Research（新加坡科技研究局）；Centre for Frontier AI Research（前沿人工智能研究中心）；Institute for Infocomm Research（资讯通信研究院）
Contrastive Learning Meets Pseudo-label-assisted Mixup Augmentation: A Comprehensive Graph Representation Framework from Local to Global
School of Information Science and Technology, Beijing University of Technology（北京工业大学信息科学与技术学院）；The University of Sydney Business School, The University of Sydney（悉尼大学商学院）；College of Information Science and Technology, Beijing University of Chemical Technology（北京化工大学信息科学与技术学院）
Quantum-Informed Contrastive Learning with Dynamic Mixup Augmentation for Class-Imbalanced Expert Systems
organization= Thomas Lord Department of Computer Science, Viterbi School of Engineering, University of Southern California , city= Los Angeles , state= CA , postcode= 90089 , country= USA；organization= Department of Industrial Engineering；Management, Khulna University of Engineering \& Technology (KUET) , city= Khulna , postcode= 9203 , country= Bangladesh；organization= Department of Computer Science, American International University-Bangladesh (AIUB) , city= Dhaka , postcode= 1229 , country= Bangladesh
Continual Contrastive Learning on Tabular Data with Out of Distribution
Indonesia Endowment Fund for Education Agency (LPDP)（印度尼西亚教育基金机构（LPDP））；School of Electrical Engineering and Computer Science（电气工程与计算机科学学院）；The University of Queensland（昆士兰大学）；Department of Computing（计算机系）；The Hong Kong Polytechnic University（香港理工大学）
Zhejiang University（浙江大学）；Zhejiang University - University of Illinois at Urbana-Champaign Institute（浙江大学伊利诺伊大学厄巴纳香槟校区联合学院）
Class imbalance has a detrimental effect on the predictive performance of most supervised learning algorithms as the imbalanced distribution can lead to a bias preferring the majority class. To solve this problem, we propose a Supervised Contrastive Learning (SCL) method with Tree-structured Parzen Estimator (TPE) technique for imbalanced tabular datasets. Contrastive learning (CL) can extract the information hidden in data even without labels and has shown some potential for imbalanced learning tasks. SCL further considers the label information based on CL, which also addresses the insufficie…
SupReMix: Supervised Contrastive Learning for Medical Imaging Regression with Mixup
Yong Loo Lin School of Medicine, National University of Singapore（杨潞龄医学院，新加坡国立大学）；National University of Singapore（新加坡国立大学）；Duke University（杜克大学）；ETH Zurich（苏黎世联邦理工学院）
Contrastive Federated Learning with Tabular Data Silos
The University of Queensland（昆士兰大学）；Monash University（莫纳什大学）；Indonesia Taxation Office（印度尼西亚税务局）；Queensland University of Technology（昆士兰科技大学）；SUNY Albany（纽约州立大学奥尔巴尼分校）；Dartmouth College（达特茅斯学院）；Hong Kong Polytechnic University（香港理工大学）；Renmin University of China（中国人民大学）
FairContrast: Enhancing Fairness through Contrastive learning and Customized Augmenting Methods on Tabular Data
Department of Industrial Engineering University of Central Florida（工业工程系，中央佛罗里达大学）；Department of Computer Science University of Central Florida（计算机科学系，中央佛罗里达大学）
Texas A&M University（德州农工大学）；University of Georgia（佐治亚大学）；Rice University（莱斯大学）
This work develops \emph{mixup for graph data}. Mixup has shown superiority in improving the generalization and robustness of neural networks by interpolating features and labels between two random samples. Traditionally, Mixup can work on regular, grid-like, and Euclidean data such as image or tabular data. However, it is challenging to directly adopt Mixup to augment graph data because different graphs typically: 1) have different numbers of nodes; 2) are not readily aligned; and 3) have unique typologies in non-Euclidean space. To this end, we propose $\mathcal{G}$-Mixup to augment graphs f…
Contrastive self-supervised learning has been successfully used in many domains, such as images, texts, graphs, etc., to learn features without requiring label information. In this paper, we propose a new local contrastive feature learning (LoCL) framework, and our theme is to learn local patterns/features from tabular data. In order to create a niche for local learning, we use feature correlations to create a maximum-spanning tree, and break the tree into feature subsets, with strongly correlated features being assigned next to each other. Convolutional learning of the features is used to lea…
Improving the generalization of deep networks is an important open challenge, particularly in domains without plentiful data. The mixup algorithm improves generalization by linearly interpolating a pair of examples and their corresponding labels. These interpolated examples augment the original training set. Mixup has shown promising results in various classification tasks, but systematic analysis of mixup in regression remains underexplored. Using mixup directly on regression labels can result in arbitrarily incorrect labels. In this paper, we propose a simple yet powerful algorithm, C-Mixup,…
University of Maryland, College Park（马里兰大学帕克分校）；Capital One（第一资本金融公司）
Tabular data underpins numerous high-impact applications of machine learning from fraud detection to genomics and healthcare. Classical approaches to solving tabular problems, such as gradient boosting and random forests, are widely used by practitioners. However, recent deep learning methods have achieved a degree of performance competitive with popular techniques. We devise a hybrid deep learning approach to solving tabular data problems. Our method, SAINT, performs attention over both rows and columns, and it includes an enhanced embedding method. We also study a new contrastive self-superv…
Class-aware and Augmentation-free Contrastive Learning from Label Proportion
The Hong Kong University of Science and Technology（香港科技大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Tencent America（腾讯美国公司）；Tencent Europe B.V.（腾讯欧洲有限责任公司）
ID-MixGCL: Identity Mixup for Graph Contrastive Learning
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；Kuaishou（快手）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）
ZAYAN: Disentangled Contrastive Transformer for Tabular Remote Sensing Data
Lane Dept. of Computer Science and Electrical Engineering（兰恩计算机科学与电气工程系）；Dept. of Geology & Geography（地质与地理系）；West Virginia University（西弗吉尼亚大学）；Department of Computer Science and Engineering（计算机科学与工程系）；Green University of Bangladesh（孟加拉国绿色大学）；Stamford University Bangladesh（班加罗尔斯坦福大学）
Code and Pixels: Multi-Modal Contrastive Pre-training for Enhanced Tabular Data Analysis
Karolinska Institute（卡罗林斯卡学院）；Helmholtz Imaging（亥姆霍兹成像研究所）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；Institute of Materials Systems Modeling（材料系统建模研究所）；Helmholtz-Zentrum Hereon（亥姆霍兹-中心赫伦）；Institute of Membrane Research（膜研究所）
Self-supervised contrastive representation learning has proved incredibly successful in the vision and natural language domains, enabling state-of-the-art performance with orders of magnitude less labeled data. However, such methods are domain-specific and little has been done to leverage this technique on real-world tabular datasets. We propose SCARF, a simple, widely-applicable technique for contrastive learning, where views are formed by corrupting a random subset of features. When applied to pre-train deep neural networks on the 69 real-world, tabular classification datasets from the OpenM…
MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies
LAION；University of California San Diego（加州大学圣迭戈分校）；Mila, Quebec Artificial Intelligence Institute（Mila魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；University of Surrey（萨里大学）
Contrastive self-supervised learning has shown impressive results in learning visual representations from unlabeled images by enforcing invariance against different data augmentations. However, the learned representations are often contextually biased to the spurious scene correlations of different objects or object and background, which may harm their generalization on the downstream tasks. To tackle the issue, we develop a novel object-aware contrastive learning framework that first (a) localizes objects in a self-supervised manner and then (b) debias scene correlations via appropriate data …
Google Research（谷歌研究院）；Carnegie Mellon University（卡内基梅隆大学）
We consider the task of self-supervised representation learning (SSL) for tabular data: tabular-SSL. Typical contrastive learning based SSL methods require instance-wise data augmentations which are difficult to design for unstructured tabular data. Existing tabular-SSL methods design such augmentations in a relatively ad-hoc fashion and can fail to capture the underlying data manifold. Instead of augmentations based approaches for tabular-SSL, we propose a new reconstruction based method, called Masked Encoding for Tabular Data (MET), that does not require augmentations. MET is based on the p…
Negative sampling is a limiting factor w.r.t. the generalization of metric-learned neural networks. We show that uniform negative sampling provides little information about the class boundaries and thus propose three novel techniques for efficient negative sampling: drawing negative samples from (1) the top-$k$ most semantically similar classes, (2) the top-$k$ most semantically similar samples and (3) interpolating between contrastive latent representations to create pseudo negatives. Our experiments on CIFAR-10, CIFAR-100 and Tiny-ImageNet-200 show that our proposed \textit{Semantically Cond…
Despite recent success, most contrastive self-supervised learning methods are domain-specific, relying heavily on data augmentation techniques that require knowledge about a particular domain, such as image cropping and rotation. To overcome such limitation, we propose a novel domain-agnostic approach to contrastive learning, named DACL, that is applicable to domains where invariances, and thus, data augmentation techniques, are not readily available. Key to our approach is the use of Mixup noise to create similar and dissimilar examples by mixing data samples differently either at the input o…
Recent advances in self-supervised learning (SSL) using large models to learn visual representations from natural images are rapidly closing the gap between the results produced by fully supervised learning and those produced by SSL on downstream vision tasks. Inspired by this advancement and primarily motivated by the emergence of tabular and structured document image applications, we investigate which self-supervised pretraining objectives, architectures, and fine-tuning strategies are most effective. To address these questions, we introduce RegCLR, a new self-supervised framework that combi…
Despite the recent development in the topic of explainable AI/ML for image and text data, the majority of current solutions are not suitable to explain the prediction of neural network models when the datasets are tabular and their features are in high-dimensional vectorized formats. To mitigate this limitation, therefore, we borrow two notable ideas (i.e., "explanation by intervention" from causality and "explanation are contrastive" from philosophy) and propose a novel solution, named as GRACE, that better explains neural network models' predictions for tabular datasets. In particular, given…
University of Michigan（密歇根大学）；Amazon Web Services（亚马逊云科技）；Google Cloud AI（谷歌云人工智能）；KAIST（韩国科学技术院）；LG AI Research（LG人工智能研究院）
Contrastive representation learning has shown to be effective to learn representations from unlabeled data. However, much progress has been made in vision domains relying on data augmentations carefully designed using domain knowledge. In this work, we propose i-Mix, a simple yet effective domain-agnostic regularization strategy for improving contrastive representation learning. We cast contrastive learning as training a non-parametric classifier by assigning a unique virtual class to each data in a batch. Then, data instances are mixed in both the input and virtual label spaces, providing mor…
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
Contrastive representation learning seeks to acquire useful representations by estimating the shared information between multiple views of data. Here, the choice of data augmentation is sensitive to the quality of learned representations: as harder the data augmentations are applied, the views share more task-relevant information, but also task-irrelevant one that can hinder the generalization capability of representation. Motivated by this, we present a new robust contrastive learning scheme, coined RényiCL, which can effectively manage harder augmentations by utilizing Rényi divergence. Our …
Imperial College London（伦敦帝国学院）；DeepMind（深度思维）；UC Berkeley（加州大学伯克利分校）；University of Sussex（萨塞克斯大学）
Computer vision algorithms, e.g. for face recognition, favour groups of individuals that are better represented in the training data. This happens because of the generalization that classifiers have to make. It is simpler to fit the majority groups as this fit is more important to overall error. We propose to create a balanced training dataset, consisting of the original dataset plus new data points in which the group memberships are intervened, minorities become majorities and vice versa. We show that current generative adversarial networks are a powerful tool for learning these data points, …
For an image query, unsupervised contrastive learning labels crops of the same image as positives, and other image crops as negatives. Although intuitive, such a native label assignment strategy cannot reveal the underlying semantic similarity between a query and its positives and negatives, and impairs performance, since some negatives are semantically similar to the query or even share the same semantic class as the query. In this work, we first prove that for contrastive learning, inaccurate label assignment heavily impairs its generalization for semantic instance discrimination, while accu…
