In this paper, we propose a novel unsupervised clustering approach exploiting the hidden information that is indirectly introduced through a pseudo classification objective. Specifically, we randomly assign a pseudo parent-class label to each observation which is then modified by applying the domain specific transformation associated with the assigned label. Generated pseudo observation-label pairs are subsequently used to train a neural network with Auto-clustering Output Layer (ACOL) that introduces multiple softmax nodes for each pseudo parent-class. Due to the unsupervised objective based on Graph-based Activity Regularization (GAR) terms, softmax duplicates of each parent-class are specialized as the hidden information captured through the help of domain specific transformations is propagated during training. Ultimately we obtain a k-means friendly latent representation. Furthermore, we demonstrate how the chosen transformation type impacts performance and helps propagate the latent information that is useful in revealing unknown clusters. Our results show state-of-the-art performance for unsupervised clustering tasks on MNIST, SVHN and USPS datasets, with the highest accuracies reported to date in the literature.
Space Applications Centre（空间应用中心）；Indian Space Research Organization（印度空间研究组织）
Despite numerous attempts sought to provide empirical evidence of adversarial regularization outperforming sole supervision, the theoretical understanding of such phenomena remains elusive. In this study, we aim to resolve whether adversarial regularization indeed performs better than sole supervision at a fundamental level. To bring this insight into fruition, we study vanishing gradient issue, asymptotic iteration complexity, gradient flow and provable convergence in the context of sole supervision and adversarial regularization. The key ingredient is a theoretical justification supported by…
Consistency regularization on label predictions becomes a fundamental technique in semi-supervised learning, but it still requires a large number of training iterations for high performance. In this study, we analyze that the consistency regularization restricts the propagation of labeling information due to the exclusion of samples with unconfident pseudo-labels in the model updates. Then, we propose contrastive regularization to improve both efficiency and accuracy of the consistency regularization by well-clustered features of unlabeled data. In specific, after strongly augmented samples ar…
University of Turin（都灵大学）；National Institute of Astrophysics（国家天体物理研究所）；Univ. Pierre et Marie Curie（皮埃尔和玛丽·居里大学）
In this paper we consider the training of single hidden layer neural networks by pseudoinversion, which, in spite of its popularity, is sometimes affected by numerical instability issues. Regularization is known to be effective in such cases, so that we introduce, in the framework of Tikhonov regularization, a matricial reformulation of the problem which allows us to use the condition number as a diagnostic tool for identification of instability. By imposing well-conditioning requirements on the relevant matrices, our theoretical analysis allows the identification of an optimal value for the r…
In-context Demonstration Matters: On Prompt Optimization for Pseudo-Supervision Refinement
Center for Advanced Intelligence Project, RIKEN（RIKEN先进智能项目中心）；Northeastern University（东北大学）；UNC-Chapel Hill（北卡罗来纳大学教堂山分校）；Center for Advanced Intelligence Project, RIKEN Graduate School of Frontier Sciences, The University of Tokyo（RIKEN前沿科学研究生院、东京大学先进智能项目中心）
Tensor decomposition methods allow us to learn the parameters of latent variable models through decomposition of low-order moments of data. A significant limitation of these algorithms is that there exists no general method to regularize them, and in the past regularization has mostly been performed using bespoke modifications to the algorithms, tailored for the particular form of the desired regularizer. We present a general method of regularizing tensor decomposition methods which can be used for any likelihood model that is learnable using tensor decomposition methods and any differentiable…
Never Skip a Batch: Dense Learning of Temporal GNNs via Adaptive Pseudo-Supervision
Risk AI Research Lab（风险人工智能研究实验室）；Moscow Independent Research Institute of Artificial Intelligence（莫斯科独立人工智能研究所）；Applied AI Institute（应用人工智能研究所）；Beijing Institute of Mathematical Sciences and Applications（北京雁栖湖应用数学研究院）
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；NLPR, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室）
Partial-label learning (PLL) is an important branch of weakly supervised learning where the single ground truth resides in a set of candidate labels, while the research rarely considers the label imbalance. A recent study for imbalanced partial-Label learning proposed that the combinatorial challenge of partial-label learning and long-tail learning lies in matching between a decent marginal prior distribution with drawing the pseudo labels. However, we believe that even if the pseudo label matches the prior distribution, the tail classes will still be difficult to learn because the total weigh…
Weak supervision (WS) is a rich set of techniques that produce pseudolabels by aggregating easily obtained but potentially noisy label estimates from a variety of sources. WS is theoretically well understood for binary classification, where simple approaches enable consistent estimation of pseudolabel noise rates. Using this result, it has been shown that downstream models trained on the pseudolabels have generalization guarantees nearly identical to those trained on clean labels. While this is exciting, users often wish to use WS for structured prediction, where the output space consists of m…
AI Clearing, Inc.（AI清算公司）；Semantic Technology Institute, Department of Computer Science, University of Innsbruck（因斯布鲁克大学语义技术研究所）；Institute of Informatics, University of Warsaw（华沙大学信息学研究所）
We present n-CPS - a generalisation of the recent state-of-the-art cross pseudo supervision (CPS) approach for the task of semi-supervised semantic segmentation. In n-CPS, there are n simultaneously trained subnetworks that learn from each other through one-hot encoding perturbation and consistency regularisation. We also show that ensembling techniques applied to subnetworks outputs can significantly improve the performance. To the best of our knowledge, n-CPS paired with CutMix outperforms CPS and sets the new state-of-the-art for Pascal VOC 2012 with (1/16, 1/8, 1/4, and 1/2 supervised regi…
FedDSR: Federated Deep Supervision and Regularization Towards Autonomous Driving
Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电子与电气工程系）；Department of Computer Science, City University of Hong Kong（城市大学计算机科学系）；Shenzhen International Center For Industrial And Applied Mathematics, Shenzhen Research Institute of Big Data（深圳国际工业与应用数学中心、深圳大数据研究院）
University College Dublin（都柏林大学学院）；Insight Centre for Data Analytics（洞察数据分析中心）
Leveraging unlabelled data through weak or distant supervision is a compelling approach to developing more effective text classification models. This paper proposes a simple but effective data augmentation method, which leverages the idea of pseudo-labelling to select samples from noisy distant supervision annotation datasets. The result shows that the proposed method improves the accuracy of biased news detection models.
In this paper, we study an intermediate form of supervision, i.e., single-frame supervision, for temporal action localization (TAL). To obtain the single-frame supervision, the annotators are asked to identify only a single frame within the temporal window of an action. This can significantly reduce the labor cost of obtaining full supervision which requires annotating the action boundary. Compared to the weak supervision that only annotates the video-level label, the single-frame supervision introduces extra temporal action signals while maintaining low annotation overhead. To make full use o…
OmniISR: A Unified Framework for Centralized and Federated Learning via Intermediate Supervision and Regularization
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China（清华大学深圳国际研究生院，清华大学，深圳，中国）；Shenzhen Research Institute of Big Data, Shenzhen, China（深圳大数据研究院，深圳，中国）；Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China（南方科技大学计算机科学与工程系，深圳，中国）；Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong, China（香港大学电子与电气工程系，香港，中国）；Yinwang Intelligent Technology Co. Ltd., Shenzhen, China（云网智能科技有限公司，深圳，中国）
This paper introduces SelfMatch, a semi-supervised learning method that combines the power of contrastive self-supervised learning and consistency regularization. SelfMatch consists of two stages: (1) self-supervised pre-training based on contrastive learning and (2) semi-supervised fine-tuning based on augmentation consistency regularization. We empirically demonstrate that SelfMatch achieves the state-of-the-art results on standard benchmark datasets such as CIFAR-10 and SVHN. For example, for CIFAR-10 with 40 labeled examples, SelfMatch achieves 93.19% accuracy that outperforms the strong p…
Deep Supervision by Gaussian Pseudo-label-based Morphological Attention for Abdominal Aorta Segmentation in Non-Contrast CTs
Univ Rennes（雷恩大学）；CHU Rennes（雷恩大学医院）；Inserm（法国国家健康与医学研究院）；Centre de Recherche en Information Biomédicale Sino-français (CRIBs)（中法生物医学信息研究中心）；Southeast University（东南大学）
Saab AB（萨博集团）；Chalmers University of Technology（查尔姆斯理工大学）
Following the success of supervised learning, semi-supervised learning (SSL) is now becoming increasingly popular. SSL is a family of methods, which in addition to a labeled training set, also use a sizable collection of unlabeled data for fitting a model. Most of the recent successful SSL methods are based on pseudo-labeling approaches: letting confident model predictions act as training labels. While these methods have shown impressive results on many benchmark datasets, a drawback of this approach is that not all unlabeled data are used during training. We propose a new SSL algorithm, Doubl…
NanoNet: Parameter-Efficient Learning with Label-Scarce Supervision for Lightweight Text Mining Model
Zhongguancun Laboratory（中关村实验室）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；Nanyang Technological University (NTU)（南洋理工大学）；Department of Computer Science, The University of Manchester（曼彻斯特大学计算机科学系）；Department of Computer Science, University of Illinois at Chicago（伊利诺伊大学香槟分校计算机科学系）
Seoul National University（首尔国立大学）；Adobe Research（奥多比研究院）
Dataset bias is a critical challenge in machine learning since it often leads to a negative impact on a model due to the unintended decision rules captured by spurious correlations. Although existing works often handle this issue based on human supervision, the availability of the proper annotations is impractical and even unrealistic. To better tackle the limitation, we propose a simple but effective unsupervised debiasing technique. Specifically, we first identify pseudo-attributes based on the results from clustering performed in the feature embedding space even without an explicit bias att…
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；University of Freiburg（弗莱堡大学）；Bosch Research, Bosch GmbH（博世研究院（博世有限公司））；Bosch Center for AI, Bosch GmbH（博世人工智能中心（博世有限公司））
Deep neural network (DNN) based salient object detection in images based on high-quality labels is expensive. Alternative unsupervised approaches rely on careful selection of multiple handcrafted saliency methods to generate noisy pseudo-ground-truth labels. In this work, we propose a two-stage mechanism for robust unsupervised object saliency prediction, where the first stage involves refinement of the noisy pseudo labels generated from different handcrafted methods. Each handcrafted method is substituted by a deep network that learns to generate the pseudo labels. These labels are refined in…
Graph data are ubiquitous in the real world. Graph learning (GL) tries to mine and analyze graph data so that valuable information can be discovered. Existing GL methods are designed for centralized scenarios. However, in practical scenarios, graph data are usually distributed in different organizations, i.e., the curse of isolated data islands. To address this problem, we incorporate federated learning into GL and propose a general Federated Graph Learning framework FedGL, which is capable of obtaining a high-quality global graph model while protecting data privacy by discovering the global s…
TABNet: A Triplet Augmentation Self-Recovery Framework with Boundary-Aware Pseudo-Labels for Medical Image Segmentation
School of Information Science and Technology, Northwest University（信息科学与技术学院，西北大学）；Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford（生物医学工程研究所，工程科学系，牛津大学）
Fine-tuned pre-trained language models can suffer from severe miscalibration for both in-distribution and out-of-distribution (OOD) data due to over-parameterization. To mitigate this issue, we propose a regularized fine-tuning method. Our method introduces two types of regularization for better calibration: (1) On-manifold regularization, which generates pseudo on-manifold samples through interpolation within the data manifold. Augmented training with these pseudo samples imposes a smoothness regularization to improve in-distribution calibration. (2) Off-manifold regularization, which encoura…
Universitat de Barcelona（巴塞罗那大学）；Universidade Federal de Santa Catarina（圣卡塔琳娜联邦大学）；Max Planck Institute of Animal Behavior（马克斯·普朗克动物行为研究所）；Universidade Federal do Paraná（巴拉那联邦大学）；Witwatersrand University（威特沃特斯兰德大学）；Oregon State University（俄勒冈州立大学）；University of Zurich（苏黎世大学）；Universitat Autonoma de Barcelona（巴塞罗那自治大学）
Effective conservation actions require effective population monitoring. However, accurately counting animals in the wild to inform conservation decision-making is difficult. Monitoring populations through image sampling has made data collection cheaper, wide-reaching and less intrusive but created a need to process and analyse this data efficiently. Counting animals from such data is challenging, particularly when densely packed in noisy images. Attempting this manually is slow and expensive, while traditional computer vision methods are limited in their generalisability. Deep learning is the …
Institute of Computing Technology, CAS（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）
Pseudo-relevance feedback (PRF) has proven to be an effective query reformulation technique to improve retrieval accuracy. It aims to alleviate the mismatch of linguistic expressions between a query and its potential relevant documents. Existing PRF methods independently treat revised queries originating from the same query but using different numbers of feedback documents, resulting in severe query drift. Without comparing the effects of two different revisions from the same query, a PRF model may incorrectly focus on the additional irrelevant information increased in the more feedback, and t…
Segment Concealed Objects with Incomplete Supervision
Department of Biomedical Engineering, Duke University（生物医学工程系，杜克大学）；Meta；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；School of Information and Communication Engineering, Dalian University of Technology（信息与通信工程学院，大连理工大学）；Shanghai Jiao Tong University（上海交通大学）
Recent advances in domain adaptation show that deep self-training presents a powerful means for unsupervised domain adaptation. These methods often involve an iterative process of predicting on target domain and then taking the confident predictions as pseudo-labels for retraining. However, since pseudo-labels can be noisy, self-training can put overconfident label belief on wrong classes, leading to deviated solutions with propagated errors. To address the problem, we propose a confidence regularized self-training (CRST) framework, formulated as regularized self-training. Our method treats ps…
