Using gamma regression for photometric redshifts of survey galaxies
J. Elliott, R. S. de Souza, A. Krone-Martins, E. Cameron, E. E. O. Ishida, J. Hilbe
备注：
Refereed Proceeding of "The Universe of Digital Sky Surveys" conference held at the INAF - Observatory of Capodimonte, Naples, on 25th-28th November 2014, to be published in the Astrophysics and Space Science Proceedings, edited by Longo, Napolitano, Marconi, Paolillo, Iodice, 6 pages, and 1 figure
Machine learning techniques offer a plethora of opportunities in tackling big data within the astronomical community. We present the set of Generalized Linear Models as a fast alternative for determining photometric redshifts of galaxies, a set of tools not commonly applied within astronomy, despite being widely used in other professions. With this technique, we achieve catastrophic outlier rates of the order of ~1%, that can be achieved in a matter of seconds on large datasets of size ~1,000,000. To make these techniques easily accessible to the astronomical community, we developed a set of libraries and tools that are publicly available.
In this work, we are trying to extent the existing photometric redshift regression models from modeling pure photometric data back to the spectra themselves. To that end, we developed a PCA that is capable of describing the input uncertainty (including missing values) in a dimensionality reduction framework. With this "spectrum generator" at hand, we are capable of treating the redshift regression problem in a fully Bayesian framework, returning a posterior distribution over the redshift. This approach allows therefore to approach the multimodal regression problem in an adequate fashion. In ad…
This work emphasizes that heterogeneity, diversity, discontinuity, and discreteness in data is to be exploited in classification and regression problems. A global a priori model may not be desirable. For data analytics in cosmology, this is motivated by the variety of cosmological objects such as elliptical, spiral, active, and merging galaxies at a wide range of redshifts. Our aim is matching and similarity-based analytics that takes account of discrete relationships in the data. The information structure of the data is represented by a hierarchy or tree where the branch structure, rather tha…
Uniform and accurate photometric calibration plays an important role in the current and next-generation wide-field imaging surveys. Herein, we review the modern photometric calibration methods, including the classic standard star method, "hardware/observation-driven" methods (such as the Ubercalibration, Hypercalibration, and Forward Global Calibration Methods), and "software/physics-driven" methods (e.g., the Stellar Locus Regression, Stellar Locus, and Stellar Color Regression Methods). Further, we discuss their advantages, limitations, and future developments toward millimagnitude precision…
Expanding upon the work of Way and Srivastava 2006 we demonstrate how the use of training sets of comparable size continue to make Gaussian process regression (GPR) a competitive approach to that of neural networks and other least-squares fitting methods. This is possible via new large size matrix inversion techniques developed for Gaussian processes (GPs) that do not require that the kernel matrix be sparse. This development, combined with a neural-network kernel function appears to give superior results for this problem. Our best fit results for the Sloan Digital Sky Survey (SDSS) Main Galax…
We present an unsupervised machine learning approach that can be employed for estimating photometric redshifts. The proposed method is based on a vector quantization approach called Self--Organizing Mapping (SOM). A variety of photometrically derived input values were utilized from the Sloan Digital Sky Survey's Main Galaxy Sample, Luminous Red Galaxy, and Quasar samples along with the PHAT0 data set from the PHoto-z Accuracy Testing project. Regression results obtained with this new approach were evaluated in terms of root mean square error (RMSE) to estimate the accuracy of the photometric r…
University of Pittsburgh（匹兹堡大学）；Federal University of São Carlos（圣卡洛斯联邦大学）；Carnegie Mellon University（卡内基梅隆大学）；Argonne National Laboratory（阿贡国家实验室）；Ruhr-University Bochum（波鸿鲁尔大学）；German Centre for Cosmological Lensing（德国宇宙透镜中心）
Many astrophysical analyses depend on estimates of redshifts (a proxy for distance) determined from photometric (i.e., imaging) data alone. Inaccurate estimates of photometric redshift uncertainties can result in large systematic errors. However, probability distribution outputs from many photometric redshift methods do not follow the frequentist definition of a Probability Density Function (PDF) for redshift -- i.e., the fraction of times the true redshift falls between two limits $z_{1}$ and $z_{2}$ should be equal to the integral of the PDF between these limits. Previous works have used the…
Measuring distances of cosmological sources such as galaxies, stars and quasars plays an increasingly critical role in modern cosmology. Obtaining the optical spectrum and consequently calculating the redshift as a distance indicator could instantly classify these objects. As long as spectroscopic observations are not available for many galaxies and the process of measuring the redshift is time-consuming and infeasible for large samples, machine learning (ML) approaches could be applied to determine the redshifts of galaxies from different features including their photometric colors. In this p…
By combining spectroscopic data from the LAMOST DR7, SDSS DR12, and corrected photometric data from the Gaia EDR3, we apply the Stellar Color Regression (SCR; Yuan et al. 2015a) method to recalibrate the SDSS Stripe 82 standard stars catalog of Ivezić et al. (2007). With a total number of about 30,000 spectroscopically targeted stars, we have mapped out the relatively large and strongly correlated photometric zero-point errors present in the catalog, $\sim$2.5 per cent in the $u$ band and $\sim$ 1 per cent in the $griz$ bands. Our study also confirms some small but significant magnitude depend…
Photometric redshifts (photo-z) are crucial to the scientific exploitation of modern panchromatic digital surveys. In this paper we present PhotoRApToR (Photometric Research Application To Redshift): a Java/C++ based desktop application capable to solve non-linear regression and multi-variate classification problems, in particular specialized for photo-z estimation. It embeds a machine learning algorithm, namely a multilayer neural network trained by the Quasi Newton learning rule, and special tools dedicated to pre- and postprocessing data. PhotoRApToR has been successfully tested on several …
We present Stellar Locus Regression (SLR), a method of directly adjusting the instrumental broadband optical colors of stars to bring them into accord with a universal stellar color-color locus, producing accurately calibrated colors for both stars and galaxies. This is achieved without first establishing individual zeropoints for each passband, and can be performed in real-time at the telescope. We demonstrate how SLR naturally makes one wholesale correction for differences in instrumental response, for atmospheric transparency, for atmospheric extinction, and for Galactic extinction. We perf…
The second Gaia data release (DR2) delivers accurate and homogeneous photometry data of the whole sky to an exquisite quality, reaching down to the unprecedented milli-magnitude (mmag) level for the G, GRP, and GBP passbands. However, the presence of magnitude-dependent systematic effects at the 10 mmag level limits its power in scientific exploitation. In this work, using about half-million stars in common with the LAMOST DR5, we apply the spectroscopy-based stellar color regression method to calibrate the Gaia G-GRP and GBP-GRP colors. With an unprecedented precision of about 1 mmag, systema…
Four gamma-ray binaries, namely PSR B1259-63, HESS J0632+057, HD 215227 and LS I +61 303, contain compact objects orbiting around massive Be stars. The nature of the compact object is only known in the case of PSR B1259-63, but the other systems could also contain young non-accreting pulsars with relativistic winds. Around periastron passage the compact objects should produce significant changes in the structure of the Be discs due to gravitational forces and eventually by ram pressure from the putative pulsar wind. Indeed, variability in the Halpha emission line has been detected in all these…
As one of the best ground-based photometric dataset, Pan-STARRS1 (PS1) has been widely used as the reference to calibrate other surveys. In this work, we present an independent validation and re-calibration of the PS1 photometry using spectroscopic data from the LAMOST DR7 and photometric data from the corrected Gaia EDR3 with the Stellar Color Regression (SCR) method. Using per band typically a total of 1.5 million LAMOST-PS1-Gaia stars as standards, we show that the PS1 photometric calibration precisions in the $grizy$ filters are around $4\sim 5$ mmag when averaged over $20'$ regions. Howev…
We present a comparison between Gaussian processes (GPs) and artificial neural networks (ANNs) as methods for determining photometric redshifts for galaxies, given training set data. In particular, we compare their degradation in performance as the training set size is degraded in ways which might be caused by the observational limitations of spectroscopy. We find that performance with large, complete training sets is very similar, although the ANN achieves slightly smaller root mean square errors. If the size of the training set is reduced by random sampling, the RMS errors of both methods in…
Machine learning techniques offer a precious tool box for use within astronomy to solve problems involving so-called big data. They provide a means to make accurate predictions about a particular system without prior knowledge of the underlying physical processes of the data. In this article, and the companion papers of this series, we present the set of Generalized Linear Models (GLMs) as a fast alternative method for tackling general astronomical problems, including the ones related to the machine learning paradigm. To demonstrate the applicability of GLMs to inherently positive and continuo…
The explosion of data in recent years has generated an increasing need for new analysis techniques in order to extract knowledge from massive datasets. Machine learning has proved particularly useful to perform this task. Fully automatized methods have recently gathered great popularity, even though those methods often lack physical interpretability. In contrast, feature based approaches can provide both well-performing models and understandable causalities with respect to the correlations found between features and physical processes. Efficient feature selection is an essential tool to boost …
Studies of massive stars using space photometry have revealed that they commonly exhibit stochastic low-frequency (SLF) variability. This has been interpreted as being caused by internal gravity waves (IGWs) excited at the interface of convective and radiative regions within stellar interiors, such as the convective core or sub-surface convection zones. We aim to compare the properties of SLF variability in massive main-sequence stars observed by the TESS mission determined by different statistical methods, and confirm the correlation between the morphology of SLF variability and a star's loca…
In the modern galaxy surveys photometric redshifts play a central role in a broad range of studies, from gravitational lensing and dark matter distribution to galaxy evolution. Using a dataset of about 25,000 galaxies from the second data release of the Kilo Degree Survey (KiDS) we obtain photometric redshifts with five different methods: (i) Random forest, (ii) Multi Layer Perceptron with Quasi Newton Algorithm, (iii) Multi Layer Perceptron with an optimization network based on the Levenberg-Marquardt learning rule, (iv) the Bayesian Photometric Redshift model (or BPZ) and (v) a classical SED…
We combine K-Nearest Neighbors (KNN) with genetic algorithm (GA) for photometric redshift estimation of quasars, short for GeneticKNN, which is a weighted KNN approach supported by GA. This approach has two improvements compared to KNN: one is the feature weighted by GA; another is that the predicted redshift is not the redshift average of K neighbors but the weighted average of median and mean of redshifts for K neighbors, i.e. $p\times z_{median} + (1-p)\times z_{mean}$. Based on the SDSS and SDSS-WISE quasar samples, we explore the performance of GeneticKNN for photometric redshift estimati…
Upcoming astronomical surveys such as the Large Synoptic Survey Telescope (LSST) will rely on photometric classification to identify the majority of the transients and variables that they discover. We present a set of techniques for photometric classification that can be applied even when the training set of spectroscopically-confirmed objects is heavily biased towards bright, low-redshift objects. Using Gaussian process regression to model arbitrary light curves in all bands simultaneously, we "augment" the training set by generating new versions of the original light curves covering a range …
Photometric redshifts (photo-z's) are fundamental in galaxy surveys to address different topics, from gravitational lensing and dark matter distribution to galaxy evolution. The Kilo Degree Survey (KiDS), i.e. the ESO public survey on the VLT Survey Telescope (VST), provides the unprecedented opportunity to exploit a large galaxy dataset with an exceptional image quality and depth in the optical wavebands. Using a KiDS subset of about 25,000 galaxies with measured spectroscopic redshifts, we have derived photo-z's using i) three different empirical methods based on supervised machine learning,…
The need to analyze the available large synoptic multi-band surveys drives the development of new data-analysis methods. Photometric redshift estimation is one field of application where such new methods improved the results, substantially. Up to now, the vast majority of applied redshift estimation methods have utilized photometric features. We aim to develop a method to derive probabilistic photometric redshift directly from multi-band imaging data, rendering pre-classification of objects and feature extraction obsolete. A modified version of a deep convolutional network was combined with a …
Correlating BASS DR3 catalogue with ALLWISE database, the data from optical and infrared information are obtained. The quasars from SDSS are taken as training and test samples while those from LAMOST are considered as external test sample. We propose two schemes to construct the redshift estimation models with XGBoost, CatBoost and Random forest. One scheme (namely one-step model) is to predict photometric redshifts directly based on the optimal models created by these three algorithms; the other scheme (namely two-step model) is to firstly classify the data into low- and high- redshift datase…
The development of fast and accurate methods of photometric redshift estimation is a vital step towards being able to fully utilize the data of next-generation surveys within precision cosmology. In this paper we apply a specific approach to spectral connectivity analysis (SCA; Lee & Wasserman 2009) called diffusion map. SCA is a class of non-linear techniques for transforming observed data (e.g., photometric colours for each galaxy, where the data lie on a complex subset of p-dimensional space) to a simpler, more natural coordinate system wherein we apply regression to make redshift predictio…
The Baire metric induces an ultrametric on a dataset and is of linear computational complexity, contrasted with the standard quadratic time agglomerative hierarchical clustering algorithm. We apply the Baire distance to spectrometric and photometric redshifts from the Sloan Digital Sky Survey using, in this work, about half a million astronomical objects. We want to know how well the (more cos\ tly to determine) spectrometric redshifts can predict the (more easily obtained) photometric redshifts, i.e. we seek to regress the spectrometric on the photometric redshifts, and we develop a clusterwi…
We present a new approach to the problem of estimating the redshift of galaxies from photometric data. The approach uses a genetic algorithm combined with non-linear regression to model the 2SLAQ LRG data set with SDSS DR7 photometry. The genetic algorithm explores the very large space of high order polynomials while only requiring optimisation of a small number of terms. We find a $σ_{\text{rms}}=0.0408\pm 0.0006$ for redshifts in the range $0.4<z< 0.7$. These results are competitive with the current state-of-the-art but can be presented simply as a polynomial which does not require the user …
RV$\times$TESS I: Modeling Asteroseismic Signals with Simultaneous Photometry and RVs
Tsinghua University（清华大学）；University of Hawai‘i（夏威夷大学）；University of Sydney（悉尼大学）；Shanghai Jiao Tong University（上海交通大学）；Australian National University（澳大利亚国立大学）；King’s College London（伦敦国王学院）；California Institute of Technology（加州理工学院）；Carnegie Institution for Science（卡内基科学研究所）；University of Southern Queensland（南昆士兰大学）
JWST Advanced Deep Extragalactic Survey (JADES) Data Release 5: Photometric Catalog
Department of Astronomy and Astrophysics, University of California, Santa Cruz（加州大学圣克鲁兹分校天体物理学与天文学系）；Kavli Institute for Cosmology, University of Cambridge（剑桥大学卡弗里宇宙学研究所）；Cavendish Laboratory, University of Cambridge（剑桥大学卡文迪许实验室）；Steward Observatory, University of Arizona（亚利桑那大学斯图德天文台）；AURA for the European Space Agency (ESA), Space Telescope Science Institute（欧洲航天局（ESA）所属美国大学研究协会（AURA）太空望远镜科学研究所）；Centro de Astrobiología (CAB), CSIC–INTA（西班牙国家航空航天技术研究所（INTA）/西班牙国家科学研究委员会（CSIC）天体生物学中心）
In this paper, we propose a spectroscopy based Stellar Color Regression (SCR) method to perform accurate color calibration for modern imaging surveys, taking advantage of millions of stellar spectra now available. The method is straightforward, insensitive to systematic errors in the spectroscopically determined stellar atmospheric parameters, applicable to regions that are effectively covered by spectroscopic surveys, and capable of delivering an accuracy of a few millimagnitudes for color calibration. As an illustration, we have applied the method to the SDSS Stripe 82 data (Ivezic et al; I0…
AGNs are very powerful galaxies characterized by extremely bright emissions coming out from their central massive black holes. Knowing the redshifts of AGNs provides us with an opportunity to determine their distance to investigate important astrophysical problems such as the evolution of the early stars, their formation along with the structure of early galaxies. The redshift determination is challenging because it requires detailed follow-up of multi-wavelength observations, often involving various astronomical facilities. Here, we employ machine learning algorithms to estimate redshifts fro…
A fundamental challenge for wide-field imaging surveys is obtaining follow-up spectroscopic observations: there are > $10^9$ photometrically cataloged sources, yet modern spectroscopic surveys are limited to ~few x $10^6$ targets. As we approach the Large Synoptic Survey Telescope (LSST) era, new algorithmic solutions are required to cope with the data deluge. Here we report the development of a machine-learning framework capable of inferring fundamental stellar parameters (Teff, log g, and [Fe/H]) using photometric-brightness variations and color alone. A training set is constructed from a sy…
We present catalogs of calibrated photometry and spectroscopic redshifts in the Extended Groth Strip, intended for studies of photometric redshifts (photo-z's). The data includes ugriz photometry from CFHTLS and Y-band photometry from the Subaru Suprime camera, as well as spectroscopic redshifts from the DEEP2, DEEP3 and 3D-HST surveys. These catalogs incorporate corrections to produce effectively matched-aperture photometry across all bands, based upon object size information available in the catalog and Moffat profile point spread function fits. We test this catalog with a simple machine lea…
We present a robust method to estimate the redshift of galaxies using Pan-STARRS1 photometric data. Our method is an adaptation of the one proposed by Beck et al. (2016) for the SDSS Data Release 12. It uses a training set of 2313724 galaxies for which the spectroscopic redshift is obtained from SDSS, and magnitudes and colours are obtained from the Pan-STARRS1 Data Release 2 survey. The photometric redshift of a galaxy is then estimated by means of a local linear regression in a 5-dimensional magnitude and colour space. Our method achieves an average bias of $\overline{Δz_{\rm norm}}=-2.01 \t…
