Machine learning techniques offer a precious tool box for use within astronomy to solve problems involving so-called big data. They provide a means to make accurate predictions about a particular system without prior knowledge of the underlying physical processes of the data. In this article, and the companion papers of this series, we present the set of Generalized Linear Models (GLMs) as a fast alternative method for tackling general astronomical problems, including the ones related to the machine learning paradigm. To demonstrate the applicability of GLMs to inherently positive and continuous physical observables, we explore their use in estimating the photometric redshifts of galaxies from their multi-wavelength photometry. Using the gamma family with a log link function we predict redshifts from the PHoto-z Accuracy Testing simulated catalogue and a subset of the Sloan Digital Sky Survey from Data Release 10. We obtain fits that result in catastrophic outlier rates as low as ~1% for simulated and ~2% for real data. Moreover, we can easily obtain such levels of precision within a matter of seconds on a normal desktop computer and with training sets that contain merely thousands of galaxies. Our software is made publicly available as an user-friendly package developed in Python, R and via an interactive web application (https://cosmostatisticsinitiative.shinyapps.io/CosmoPhotoz). This software allows users to apply a set of GLMs to their own photometric catalogues and generates publication quality plots with minimum effort from the user. By facilitating their ease of use to the astronomical community, this paper series aims to make GLMs widely known and to encourage their implementation in future large-scale projects, such as the Large Synoptic Survey Telescope.
Revealing hidden patterns in astronomical data is often the path to fundamental scientific breakthroughs; meanwhile the complexity of scientific inquiry increases as more subtle relationships are sought. Contemporary data analysis problems often elude the capabilities of classical statistical techniques, suggesting the use of cutting edge statistical methods. In this light, astronomers have overlooked a whole family of statistical techniques for exploratory data analysis and robust regression, the so-called Generalized Linear Models (GLMs). In this paper -- the first in a series aimed at illus…
One of the consequences of entering the era of precision cosmology is the widespread adoption of photometric redshift probability density functions (PDFs). Both current and future photometric surveys are expected to obtain images of billions of distinct galaxies. As a result, storing and analyzing all of these PDFs will be non-trivial and even more severe if a survey plans to compute and store multiple different PDFs. In this paper we propose the use of a sparse basis representation to fully represent individual photo-$z$ PDFs. By using an Orthogonal Matching Pursuit algorithm and a combinatio…
In this paper, the third in a series illustrating the power of generalized linear models (GLMs) for the astronomical community, we elucidate the potential of the class of GLMs which handles count data. The size of a galaxy's globular cluster population $N_{\rm GC}$ is a prolonged puzzle in the astronomical literature. It falls in the category of count data analysis, yet it is usually modelled as if it were a continuous response variable. We have developed a Bayesian negative binomial regression model to study the connection between $N_{\rm GC}$ and the following galaxy properties: central blac…
Astronomy is entering an era of data-driven discovery, due in part to modern machine learning (ML) techniques enabling powerful new ways to interpret observations. This shift in our scientific approach requires us to consider whether we can trust the black box. Here, we overview methods for an often-overlooked step in the development of ML models: building community trust in the algorithms. Trust is an essential ingredient not just for creating more robust data analysis techniques, but also for building confidence within the astronomy community to embrace machine learning methods and results.
In the last decade a new generation of telescopes and sensors has allowed the production of a very large amount of data and astronomy has become a data-rich science. New automatic methods largely based on machine learning are needed to cope with such data tsunami. We present some results in the fields of photometric redshifts and galaxy classification, obtained using the MLPQNA algorithm available in the DAMEWARE (Data Mining and Web Application Resource) for the SDSS galaxies (DR9 and DR10). We present PhotoRApToR (Photometric Research Application To Redshift): a Java based desktop applicatio…
Data challenges are emerging as powerful tools with which to answer fundamental astronomical questions. Time-domain astronomy lends itself to data challenges, particularly in the era of classification and anomaly detection. With improved sensitivity of wide-field surveys in optical and radio wavelengths from surveys like the Large Synoptic Survey Telescope (LSST) and the Canadian Hydrogen Intensity Mapping Experiment (CHIME), we are entering the large-volume era of transient astronomy. I highlight some recent time-domain challenges, with particular focus on the Photometric LSST Astronomical Ti…
Measuring photometric brightness is a common tool for characterizaing satellites. However, characterizing satellite mega-constellations and their impact on astronomy research requires a new approach and methodology. A few measurements of singular satellites are not sufficient to fully describe a mega-constellation and assess its impact on modern astronomical systems. Characterizing the brightness of a satellite mega-constellation requires a comprehensive measurement program conducting numerous observations over the entire set of critical variables. Utilizing Pomenis, a small-aperture and wide …
Photometric redshifts play an important role as a measure of distance for various cosmological topics. Spectroscopic redshifts are only available for a very limited number of objects but can be used for creating statistical models. A broad variety of photometric catalogues provide uncertain low resolution spectral information for galaxies and quasars that can be used to infer a redshift. Many different techniques have been developed to produce those redshift estimates with increasing precision. Instead of providing a point estimate only, astronomers start to generate probabilistic density func…
The Impact of Artificial Space Objects on Optical Astronomy as Determined from 32 Million Photometric Observations
IAU Centre for the Protection of the Dark and Quiet Sky（国际天文联合会保护黑暗与宁静天空中心）；Czech Academy of Sciences（捷克科学院）；Institute of Physics, Czech Academy of Sciences（捷克科学院物理研究所）
The need for accurate photometric redshifts estimation is a topic that has fundamental importance in Astronomy, due to the necessity of efficiently obtaining redshift information without the need of spectroscopic analysis. We propose a method for determining accurate multimodal photo-z probability density functions (PDFs) using Mixture Density Networks (MDN) and Deep Convolutional Networks (DCN). A comparison with a Random Forest (RF) is performed.
Department of Computer Science, University of Houston（休斯顿大学计算机科学系）；University of Houston（休斯顿大学）
The widespread dissemination of machine learning tools in science, particularly in astronomy, has revealed the limitation of working with simple single-task scenarios in which any task in need of a predictive model is looked in isolation, and ignores the existence of other similar tasks. In contrast, a new generation of techniques is emerging where predictive models can take advantage of previous experience to leverage information from similar tasks. The new emerging area is referred to as transfer learning. In this paper, I briefly describe the motivation behind the use of transfer learning t…
PhoPS: An automated photometric pipeline for survey-era astronomy
Türkiye National Observatories, TUG（土耳其国家天文台）；The Scientific and Technological Research Council of Türkiye (TÜBİTAK)（土耳其科学技术研究委员会）；Instituto de Astrofísica de Andalucía (IAA-CSIC)（安达卢西亚天体物理研究所）；LIRA, CNRS UMR8254, Observatoire de Paris（巴黎天文台）
Preliminary Report on Mantis Shrimp: a Multi-Survey Computer Vision Photometric Redshift Model
Pacific Northwest National Laboratory（西北太平洋国家实验室）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；National Center for Supercomputing Applications（国家超级计算应用中心）；Illinois Center for Advanced Studies of the Universe（伊利诺伊大学宇宙高级研究中心）
Astronomy has entered the big data era and Machine Learning based methods have found widespread use in a large variety of astronomical applications. This is demonstrated by the recent huge increase in the number of publications making use of this new approach. The usage of machine learning methods, however is still far from trivial and many problems still need to be solved. Using the evaluation of photometric redshifts as a case study, we outline the main problems and some ongoing efforts to solve them.
We apply a combination of a Genetic Algorithms (GA) and Support Vector Machines (SVM) machine learning algorithm to solve two important problems faced by the astronomical community: star/galaxy separation, and photometric redshift estimation of galaxies in survey catalogs. We use the GA to select the relevant features in the first step, followed by optimization of SVM parameters in the second step to obtain an optimal set of parameters to classify or regress, in process of which we avoid over-fitting. We apply our method to star/galaxy separation in Pan-STARRS1 data. We show that our method co…
ImageJ is a graphical user interface (GUI) driven, public domain, Java-based, software package for general image processing traditionally used mainly in life sciences fields. The image processing capabilities of ImageJ are useful and extendable to other scientific fields. Here we present AstroImageJ (AIJ), which provides an astronomy specific image display environment and tools for astronomy specific image calibration and data reduction. Although AIJ maintains the general purpose image processing capabilities of ImageJ, AIJ is streamlined for time-series differential photometry, light curve de…
PESIT South Campus（PESIT南校区）；University of Calcutta（加尔各答大学）
We present the results of various automated classification methods, based on machine learning (ML), of objects from data releases 6 and 7 (DR6 and DR7) of the Sloan Digital Sky Survey (SDSS), primarily distinguishing stars from quasars. We provide a careful scrutiny of approaches available in the literature and have highlighted the pitfalls in those approaches based on the nature of data used for the study. The aim is to investigate the appropriateness of the application of certain ML methods. The manuscript argues convincingly in favor of the efficacy of asymmetric AdaBoost to classify photom…
ImageJ is a graphical user interface (GUI) driven, public domain, Java-based, software package for general image processing traditionally used mainly in life sciences fields. The image processing capabilities of ImageJ are useful and extendable to other scientific fields. Here we present AstroImageJ (AIJ), which provides an astronomy specific image display environment and tools for astronomy specific image calibration and data reduction. Although AIJ maintains the general purpose image processing capabilities of ImageJ, AIJ is streamlined for time-series differential photometry, light curve de…
Machine learning techniques offer a plethora of opportunities in tackling big data within the astronomical community. We present the set of Generalized Linear Models as a fast alternative for determining photometric redshifts of galaxies, a set of tools not commonly applied within astronomy, despite being widely used in other professions. With this technique, we achieve catastrophic outlier rates of the order of ~1%, that can be achieved in a matter of seconds on large datasets of size ~1,000,000. To make these techniques easily accessible to the astronomical community, we developed a set of l…
We describe the application of data mining algorithms to research problems in astronomy. We posit that data mining has always been fundamental to astronomical research, since data mining is the basis of evidence-based discovery, including classification, clustering, and novelty discovery. These algorithms represent a major set of computational tools for discovery in large databases, which will be increasingly essential in the era of data-intensive astronomy. Historical examples of data mining in astronomy are reviewed, followed by a discussion of one of the largest data-producing projects anti…
In this paper we apply ideas from information theory to create a method for the design of optimal filters for photometric redshift estimation. We show the method applied to a series of simple example filters in order to motivate an intuition for how photometric redshift estimators respond to the properties of photometric passbands. We then design a realistic set of six filters covering optical wavelengths that optimize photometric redshifts for $z <= 2.3$ and $i < 25.3$. We create a simulated catalog for these optimal filters and use our filters with a photometric redshift estimation code to s…
Throughout the processing and analysis of survey data, a ubiquitous issue nowadays is that we are spoilt for choice when we need to select a methodology for some of its steps. The alternative methods usually fail and excel in different data regions, and have various advantages and drawbacks, so a combination that unites the strengths of all while suppressing the weaknesses is desirable. We propose to use a two-level hierarchy of learners. Its first level consists of training and applying the possible base methods on the first part of a known set. At the second level, we feed the output probabi…
Redshift measurement has always been a constant need in modern astronomy and cosmology. And as new surveys have been providing an immense amount of data on astronomical objects, the need to process such data automatically proves to be increasingly necessary. In this article, we use simulated data from the Dark Energy Survey, and from a pipeline originally created to classify supernovae, we developed a linear regression algorithm optimized through novel automated machine learning (AutoML) frameworks achieving an error score better than ordinary data pre-processing methods when compared with oth…
Next-generation surveys like the Legacy Survey of Space and Time (LSST) on the Vera C. Rubin Observatory will generate orders of magnitude more discoveries of transients and variable stars than previous surveys. To prepare for this data deluge, we developed the Photometric LSST Astronomical Time-series Classification Challenge (PLAsTiCC), a competition which aimed to catalyze the development of robust classifiers under LSST-like conditions of a non-representative training set for a large photometric test set of imbalanced classes. Over 1,000 teams participated in PLAsTiCC, which was hosted in …
We present new fully-automatic classification model to select extragalactic objects within astronomy photometric catalogs. Construction of the our classification model is based on the three important procedures: 1) data representation to create feature space; 2) building hypersurface in feature space to limit range of features (outliers detection); 3) building hyperplane separating extragalactic objects from the galactic ones. We trained our model with 1.7 million objects (1.4 million galaxies and quasars, 0.3 million stars). The application of the model is presented as a photometric catalog o…
We present a new automatic tool for time-domain astronomy - the Cambridge Photometric Calibration Server 2.0 - developed under OPTICON H2020 programme. It has been designed to respond to the need of automated rapid photometric data calibration and dissemination for transient events, primarily from Gaia space mission. CPCS has been in operation since 2013 and has been used to calibrate around 130 000 observations of hundreds of transients. We present the status of this tool's development and demonstrate improvements made in the second version. The tests present the ability to combine CCD imagin…
Many estimation problems in astrophysics are highly complex, with high-dimensional, non-standard data objects (e.g., images, spectra, entire distributions, etc.) that are not amenable to formal statistical analysis. To utilize such data and make accurate inferences, it is crucial to transform the data into a simpler, reduced form. Spectral kernel methods are non-linear data transformation methods that efficiently reveal the underlying geometry of observable data. Here we focus on one particular technique: diffusion maps or more generally spectral connectivity analysis (SCA). We give examples o…
Scientific Validation of the SPARC4 Pipeline: Multi-band Imaging, Polarimetry, and Photometric Time Series for Improved Characterization of Transiting Exoplanets
We present an application of a particular machine-learning method (Boosted Decision Trees, BDTs using AdaBoost) to separate stars and galaxies in photometric images using their catalog characteristics. BDTs are a well established machine learning technique used for classification purposes. They have been widely used specially in the field of particle and astroparticle physics, and we use them here in an optical astronomy application. This algorithm is able to improve from simple thresholding cuts on standard separation variables that may be affected by local effects such as blending, badly cal…
FPCA-Enhanced Simulation-Based Inference for Robust Type Ia Supernova Cosmology
Baylor University（贝勒大学）；Department of Physics and Astronomy, Texas A&M University（德克萨斯农工大学物理与天文系）；George P. and Cynthia Woods Mitchell Institute for Fundamental Physics and Astronomy, Texas A&M University（德克萨斯农工大学乔治·P·辛西娅·伍兹基础物理与天文学研究所）
It is well known in astronomy that propagating non-Gaussian prediction uncertainty in photometric redshift estimates is key to reducing bias in downstream cosmological analyses. Similarly, likelihood-free inference approaches, which are beginning to emerge as a tool for cosmological analysis, require a characterization of the full uncertainty landscape of the parameters of interest given observed data. However, most machine learning (ML) or training-based methods with open-source software target point prediction or classification, and hence fall short in quantifying uncertainty in complex regr…
Stellar flares are an important aspect of magnetic activity -- both for stellar evolution and circumstellar habitability viewpoints - but automatically and accurately finding them is still a challenge to researchers in the Big Data era of astronomy. We present an experiment to detect flares in space-borne photometric data using deep neural networks. Using a set of artificial data and real photometric data we trained a set of neural networks, and found that the best performing architectures were the recurrent neural networks (RNNs) using Long Short-Term Memory (LSTM) layers. The best trained ne…
We present an application of unsupervised machine learning - the self-organised map (SOM) - as a tool for visualising, exploring and mining the catalogues of large astronomical surveys. Self-organisation culminates in a low-resolution representation of the 'topology' of a parameter volume, and this can be exploited in various ways pertinent to astronomy. Using data from the Cosmological Evolution Survey (COSMOS), we demonstrate two key astronomical applications of the SOM: (i) object classification and selection, using the example of galaxies with active galactic nuclei as a demonstration, and…
An amateur astronomer in the modern world has the opportunity not only to make visual observations for own interest, but can make scientific astronomical observations and new discoveries in astronomy. In my example, as amateur astronomer and only through self-education, I inform about my discoveries: of the possible dwarf nova on the old digitized photographic plates and of new variable stars from sky surveys data by means of data mining; how I discovered (in the images of the sky surveys): astronomical transients, supernovae, planetary nebula candidates and new binary systems in the data of G…
Modern astronomy relies on massive databases collected by robotic telescopes and digital sky surveys, acquiring data in a much faster pace than what manual analysis can support. Among other data, these sky surveys collect information about millions and sometimes billions of extra-galactic objects. Since the very large number of objects makes manual observation impractical, automatic methods that can analyze and annotate extra-galactic objects are required to fully utilize the discovery power of these databases. Machine learning methods for annotation of celestial objects can be separated broad…
Astronomy is undergoing through a methodological revolution triggered by an unprecedented wealth of complex and accurate data. The new panchromatic, synoptic sky surveys require advanced tools for discovering patterns and trends hidden behind data which are both complex and of high dimensionality. We present DAMEWARE (DAta Mining & Exploration Web Application REsource): a general purpose, web-based, distributed data mining environment developed for the exploration of large datasets, and finely tuned for astronomical applications. By means of graphical user interfaces, it allows the user to per…
With the advent of large-scale photometric surveys of the sky, modern science witnesses the dawn of big data astronomy, where automatic handling and discovery are paramount. In this context, classification tasks are among the key capabilities a data reduction pipeline must possess in order to compile reliable datasets, to accomplish data processing with an efficiency level impossible to achieve by means of detailed processing and human intervention. The VISTA Variables of the Vía Láctea Survey, in the southern part of the Galactic disc, comprises multi-epoch photometric data necessary for the …
