Sky surveys are the largest data generators in astronomy, making automated tools for extracting meaningful scientific information an absolute necessity. We show that, without the need for labels, self-supervised learning recovers representations of sky survey images that are semantically useful for a variety of scientific tasks. These representations can be directly used as features, or fine-tuned, to outperform supervised methods trained only on labeled data. We apply a contrastive learning framework on multi-band galaxy photometry from the Sloan Digital Sky Survey (SDSS) to learn image representations. We then use them for galaxy morphology classification, and fine-tune them for photometric redshift estimation, using labels from the Galaxy Zoo 2 dataset and SDSS spectroscopy. In both downstream tasks, using the same learned representations, we outperform the supervised state-of-the-art results, and we show that our approach can achieve the accuracy of supervised models while using 2-4 times fewer labels for training.
This paper briefly reviews the connections between meta-learning and self-supervised learning. Meta-learning can be applied to improve model generalization capability and to construct general AI algorithms. Self-supervised learning utilizes self-supervision from original data and extracts higher-level generalizable features through unsupervised pre-training or optimization of contrastive loss objectives. In self-supervised learning, data augmentation techniques are widely applied and data labels are not required since pseudo labels can be estimated from trained models on similar tasks. Meta-le…
In this work, we observe that many existing self-supervised learning algorithms can be both unified and generalized when seen through the lens of equivariant representations. Specifically, we introduce a general framework we call Homomorphic Self-Supervised Learning, and theoretically show how it may subsume the use of input-augmentations provided an augmentation-homomorphic feature extractor. We validate this theory experimentally for simple augmentations, demonstrate how the framework fails when representational structure is removed, and further empirically explore how the parameters of this…
This paper proposes a novel self-supervised learning method for learning better representations with small batch sizes. Many self-supervised learning methods based on certain forms of the siamese network have emerged and received significant attention. However, these methods need to use large batch sizes to learn good representations and require heavy computational resources. We present a new triplet network combined with a triple-view loss to improve the performance of self-supervised representation learning with small batch sizes. Experimental results show that our method can drastically out…
Facebook AI Research（Facebook AI研究院）；Inria（法国国家信息与自动化研究所）；Univ. Grenoble Alpes（格勒诺布尔大学）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）；LJK（让·昆塔纳实验室）
Recently, self-supervised learning methods like MoCo, SimCLR, BYOL and SwAV have reduced the gap with supervised methods. These results have been achieved in a control environment, that is the highly curated ImageNet dataset. However, the premise of self-supervised learning is that it can learn from any random image and from any unbounded dataset. In this work, we explore if self-supervision lives to its expectation by training large models on random, uncurated images with no supervision. Our final SElf-supERvised (SEER) model, a RegNetY with 1.3B parameters trained on 1B random images with 51…
Recently self supervised learning has seen explosive growth and use in variety of machine learning tasks because of its ability to avoid the cost of annotating large-scale datasets. This paper gives an overview for best self supervised learning approaches for multimodal learning. The presented approaches have been aggregated by extensive study of the literature and tackle the application of self supervised learning in different ways. The approaches discussed are cross modal generation, cross modal pretraining, cyclic translation, and generating unimodal labels in self supervised fashion.
In vision-based reinforcement learning (RL) tasks, it is prevalent to assign auxiliary tasks with a surrogate self-supervised loss so as to obtain more semantic representations and improve sample efficiency. However, abundant information in self-supervised auxiliary tasks has been disregarded, since the representation learning part and the decision-making part are separated. To sufficiently utilize information in auxiliary tasks, we present a simple yet effective idea to employ self-supervised loss as an intrinsic reward, called Intrinsically Motivated Self-Supervised learning in Reinforcement…
In observational astronomy, noise obscures signals of interest. Large-scale astronomical surveys are growing in size and complexity, which will produce more data and increase the workload of data processing. Developing automated tools, such as convolutional neural networks (CNN), for denoising has become a promising area of research. We investigate the feasibility of CNN-based self-supervised learning algorithms (e.g., Noise2Noise) for denoising astronomical images. We experimented with Noise2Noise on simulated noisy astronomical data. We evaluate the results based on the accuracy of recoverin…
Tsinghua University（清华大学）；Beijing University of Chemical Technology（北京化工大学）；Singapore Management University（新加坡管理大学）；Information Research Center of Military Science（军事科学院信息研究中心）；China Mobile Research Institute（中国移动研究院）
Self-supervised entity alignment (EA) aims to link equivalent entities across different knowledge graphs (KGs) without seed alignments. The current SOTA self-supervised EA method draws inspiration from contrastive learning, originally designed in computer vision based on instance discrimination and contrastive loss, and suffers from two shortcomings. Firstly, it puts unidirectional emphasis on pushing sampled negative entities far away rather than pulling positively aligned pairs close, as is done in the well-established supervised EA. Secondly, KGs contain rich side information (e.g., entity …
Arizona State University（亚利桑那州立大学）；Microsoft Corporation（微软公司）
This paper is concerned with self-supervised learning for small models. The problem is motivated by our empirical studies that while the widely used contrastive self-supervised learning method has shown great progress on large model training, it does not work well for small models. To address this problem, we propose a new learning paradigm, named SElf-SupErvised Distillation (SEED), where we leverage a larger network (as Teacher) to transfer its representational knowledge into a smaller architecture (as Student) in a self-supervised fashion. Instead of directly learning from unlabeled data, w…
University of Tübingen（图宾根大学）；IMPRS-IS（国际马克斯·普朗克研究所以及系统生物学研究所(IMPRS-IS) (注：IMPRS-IS通常指Max Planck Institute for Intelligent Systems, 即马克斯·普朗克智能系统研究所，但根据常见中文译法及上下文，此处保留缩写更准确)）
How do humans learn to acquire a powerful, flexible and robust representation of objects? While much of this process remains unknown, it is clear that humans do not require millions of object labels. Excitingly, recent algorithmic advancements in self-supervised learning now enable convolutional neural networks (CNNs) to learn useful visual object representations without supervised labels, too. In the light of this recent breakthrough, we here compare self-supervised networks to supervised models and human behaviour. We tested models on 15 generalisation datasets for which large-scale human be…
Recently, learning from vast unlabeled data, especially self-supervised learning, has been emerging and attracted widespread attention. Self-supervised learning followed by the supervised fine-tuning on a few labeled examples can significantly improve label efficiency and outperform standard supervised training using fully annotated data. In this work, we present a novel self-supervised deep learning paradigm based on online hard negative pair mining. Specifically, we design a student-teacher network to generate multi-view of the data for self-supervised learning and integrate hard negative pa…
Self-supervised learning aims to learn representations from the data itself without explicit manual supervision. Existing efforts ignore a crucial aspect of self-supervised learning - the ability to scale to large amount of data because self-supervision requires no manual labels. In this work, we revisit this principle and scale two popular self-supervised approaches to 100 million images. We show that by scaling on various axes (including data size and problem 'hardness'), one can largely match or even exceed the performance of supervised pre-training on a variety of tasks such as object dete…
We present a plug-in replacement for batch normalization (BN) called exponential moving average normalization (EMAN), which improves the performance of existing student-teacher based self- and semi-supervised learning techniques. Unlike the standard BN, where the statistics are computed within each batch, EMAN, used in the teacher, updates its statistics by exponential moving average from the BN statistics of the student. This design reduces the intrinsic cross-sample dependency of BN and enhances the generalization of the teacher. EMAN improves strong baselines for self-supervised learning by…
We present an electrocardiogram (ECG) -based emotion recognition system using self-supervised learning. Our proposed architecture consists of two main networks, a signal transformation recognition network and an emotion recognition network. First, unlabelled data are used to successfully train the former network to detect specific pre-determined signal transformations in the self-supervised learning step. Next, the weights of the convolutional layers of this network are transferred to the emotion recognition network, and two dense layers are trained in order to classify arousal and valence sco…
Self-supervised representation learning has seen remarkable progress in the last few years. More recently, contrastive instance learning has shown impressive results compared to its supervised learning counterparts. However, even with the ever increased interest in contrastive instance learning, it is still largely unclear why these methods work so well. In this paper, we aim to unravel some of the mysteries behind their success, which are the good practices. Through an extensive empirical analysis, we hope to not only provide insights but also lay out a set of best practices that led to the s…
Semi-supervised Source Detection in Astronomical Images: New Benchmark and Strong Baseline
School of Software Dalian University of Technology（大连理工大学软件学院）；National Astronomical Observatories Chinese Academy of Sciences（中国科学院国家天文台）；Research Institute of Highway Ministry of Transport（交通运输部公路科学研究院）
Deeper detection limits in astronomical imaging using self-supervised spatiotemporal denoising
Department of Automation, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China（自动化系，北京信息科学与技术国家研究中心，清华大学，北京，中国）；Department of Astronomy, Tsinghua University, Beijing, China（天文系，清华大学，北京，中国）；Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China（脑与认知科学研究院，清华大学，北京，中国）；Kavli Institute for the Physics and Mathematics of the Universe (WPI), The University of Tokyo, Kashiwa, Japan（宇宙物理与数学科学研究所（WPI），东京大学，日光，日本）；National Astronomical Observatory of Japan, Tokyo, Japan（日本国家天文台，东京，日本）；Institute for Cosmic Ray Research, The University of Tokyo, Kashiwa, Japan（宇宙射线研究所，东京大学，日光，日本）；College of AI, Tsinghua University, Beijing, China（人工智能学院，清华大学，北京，中国）；School of Mathematics and Physics, Qinghai University, Xining, China（数学与物理学院，青海大学，西宁，中国）；Tsinghua Wuxi Institute of Applied Technology, Wuxi, China（清华大学无锡应用技术研究院，无锡，中国）
SIFARI: Self-Supervised Interferometric Fitting for Astronomical Radio Imaging
Rice University（莱斯大学）；Rice Space Institute（莱斯空间研究所）；National Radio Astronomy Observatory（美国国家射电天文台）；NSF-Simons AI Institute for Cosmic Origins（NSF-西蒙斯宇宙起源人工智能研究所）；University of Pennsylvania（宾夕法尼亚大学）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）
We have developed a method that maps large astronomical images onto a two-dimensional map and clusters them. A combination of various state-of-the-art machine learning (ML) algorithms is used to develop a fully unsupervised image quality assessment and clustering system. Our pipeline consists of a data pre-processing step where individual image objects are identified in a large astronomical image and converted to smaller pixel images. This data is then fed to a deep convolutional autoencoder jointly trained with a self-organizing map (SOM). This part can be used as a recommendation system. The…
The Millennium Institute of Astrophysics（千年天体物理研究所）；Pontificia Universidad Católica de Chile（智利天主教大学）；Harvard University（哈佛大学）；Harvard-Smithsonian Center for Astrophysics（哈佛-史密森天体物理中心）；Max-Planck Institute for Astronomy（马克斯·普朗克天文研究所）
The development of synoptic sky surveys has led to a massive amount of data for which resources needed for analysis are beyond human capabilities. To process this information and to extract all possible knowledge, machine learning techniques become necessary. Here we present a new method to automatically discover unknown variable objects in large astronomical catalogs. With the aim of taking full advantage of all the information we have about known objects, our method is based on a supervised algorithm. In particular, we train a random forest classifier using known variability classes of objec…
The radio astronomy community is currently building a number of phased array telescopes. The calibration of these telescopes is hampered by the fact that covariances of signals from closely spaced antennas are sensitive to noise coupling and to variations in sky brightness on large spatial scales. These effects are difficult and computationally expensive to model. We propose to model them phenomenologically using a non-diagonal noise covariance matrix. The parameters can be estimated using a weighted alternating least squares (WALS) algorithm iterating between the calibration parameters and th…
Self-Driving Telescopes: Autonomous Scheduling of Astronomical Observation Campaigns with Offline Reinforcement Learning
University of Pisa（比萨大学）；Fermi National Accelerator Laboratory（费米国家加速器实验室）；Kavli Institute for Cosmological Physics（卡夫利宇宙物理研究所）；University of Chicago（芝加哥大学）；Space Telescope Science Institute（太空望远镜科学研究所）
Universidad de Chile（智利大学）；Harvard University（哈佛大学）；Millennium Institute of Astrophysics（千年天体物理研究所）
Due to the latest advances in technology, telescopes with significant sky coverage will produce millions of astronomical alerts per night that must be classified both rapidly and automatically. Currently, classification consists of supervised machine learning algorithms whose performance is limited by the number of existing annotations of astronomical objects and their highly imbalanced class distributions. In this work, we propose a data augmentation methodology based on Generative Adversarial Networks (GANs) to generate a variety of synthetic light curves from variable stars. Our novel contr…
Taiyuan University of Technology（太原理工大学）；Durham University（杜伦大学）
Image restoration methods are commonly used to improve the quality of astronomical images. In recent years, developments of deep neural networks and increments of the number of astronomical images have evoked a lot of data--driven image restoration methods. However, most of these methods belong to supervised learning algorithms, which require paired images either from real observations or simulated data as training set. For some applications, it is hard to get enough paired images from real observations and simulated images are quite different from real observed ones. In this paper, we propose…
Universidad de los Andes（洛斯安第斯大学）；Center for Research and Formation in Artificial Intelligence（人工智能研究与培训中心）；Systems and Computing Engineering Department（系统与计算工程系）；Departamento de Física（物理系）
Supervised classification of temporal sequences of astronomical images into meaningful transient astrophysical phenomena has been considered a hard problem because it requires the intervention of human experts. The classifier uses the expert's knowledge to find heuristic features to process the images, for instance, by performing image subtraction or by extracting sparse information such as flux time series, also known as light curves. We present a successful deep learning approach that learns directly from imaging data. Our method models explicitly the spatio-temporal patterns with Deep Convo…
Nanjing University of Posts and Telecommunications（南京邮电大学）；National Astronomical Observatories, Chinese Academy of Sciences（中国科学院国家天文台）；University of Chinese Academy of Sciences（中国科学院大学）
Astronomical outliers, such as unusual, rare or unknown types of astronomical objects or phenomena, constantly lead to the discovery of genuinely unforeseen knowledge in astronomy. More unpredictable outliers will be uncovered in principle with the increment of the coverage and quality of upcoming survey data. However, it is a severe challenge to mine rare and unexpected targets from enormous data with human inspection due to a significant workload. Supervised learning is also unsuitable for this purpose since designing proper training sets for unanticipated signals is unworkable. Motivated by…
BGRem: A background noise remover for astronomical images based on a diffusion model
Radboud University Nijmegen（奈梅亨拉德堡德大学）；Nikhef（荷兰国家核子与粒子物理研究所）；University of Oxford（牛津大学）；University of Nova Gorica（新戈里察大学）；Instituto de Física Corpuscular (IFIC), CSIC-UV（瓦伦西亚-西班牙国家科学研究委员会粒子物理研究所）；University of Cape Town（开普敦大学）；South African Astronomical Observatory（南非天文台）
AstroClearNet: Deep image prior for multi-frame astronomical image restoration
Department of Applied Mathematics and Statistics, Johns Hopkins University（应用数学与统计学系，约翰霍普金斯大学）；Johns Hopkins University（约翰霍普金斯大学）；Department of Computer Science, Johns Hopkins University（计算机科学系，约翰霍普金斯大学）；Department of Physics and Astronomy, Johns Hopkins University（物理与天文学系，约翰霍普金斯大学）
Machine-learning based classifiers have become indispensable in the field of astrophysics, allowing separation of astronomical sources into various classes, with computational efficiency suitable for application to the enormous data volumes that wide-area surveys now typically produce. In the standard supervised classification paradigm, a model is typically trained and validated using data from relatively small areas of sky, before being used to classify sources in other areas of the sky. However, population shifts between the training examples and the sources to be classified can lead to `sil…
