We present the design of a novel way of handling astronomical catalogs in Astro-WISE in order to achieve the scalability required for the data produced by large scale surveys. A high level of automation and abstraction is achieved in order to facilitate interoperation with visualization software for interactive exploration. At the same time flexibility in processing is enhanced and data is shared implicitly between scientists.
This is accomplished by using a data model that primarily stores how catalogs are derived; the contents of the catalogs are only created when necessary and stored only when beneficial for performance. Discovery of existing catalogs and creation of new catalogs is done through the same process by directly requesting the final set of sources (astronomical objects) and attributes (physical properties) that is required, for example from within visualization software.
New catalogs are automatically created to provide attributes of sources for which no suitable existing catalogs can be found. These catalogs are defined to contain the new attributes on the largest set of sources the calculation of the attributes is applicable to, facilitating reuse for future data requests. Subsequently, only those parts of the catalogs that are required for the requested end product are actually processed, ensuring scalability.
The presented mechanisms primarily determine which catalogs are created and what data has to be processed and stored: the actual processing and storage itself is left to existing functionality of the underlying information system.
Interactive visualization of astronomical catalogs requires novel techniques due to the huge volumes and complex structure of the data produced by existing and upcoming astronomical surveys. The creation as well as the disclosure of the catalogs can be handled by data pulling mechanisms. These prevent unnecessary processing and facilitate data sharing by having users request the desired end products. In this work we present query driven visualization as a logical continuation of data pulling. Scientists can request catalogs in a declarative way and set process parameters directly from within t…
The visual inspection of image and catalog data continues to be a valuable aspect of astronomical data analysis. As the scale of astronomical image and catalog data continues to grow, visualizing the data becomes increasingly difficult. In this work, we introduce FitsMap, a simple, lightweight tool for visualizing astronomical image and catalog data. FitsMap only requires a simple web server and can scale to over gigapixel images with tens of millions of sources. Further, the web-based visualizations can be viewed performantly on mobile devices. FitsMap is implemented in Python and is open sou…
In this paper we presented the algorithm designed to efficient coordinate cross-match of objects in the modern massive astronomical catalogues. Preliminary data sort in the existed catalogues provides the opportunity for coordinate identification of the objects without any constraints with the storage and technical environment (PC). Using the multi-threading of the modern computing processors allows speeding up the program up to read-write data to the storage. Also the paper contains the main difficulties of implementing of the algorithm, as well as their possible solutions.
Fast access to large catalogs is required for some astronomical applications. Here we introduce the catsHTM tool, consisting of several large catalogs reformatted into HDF5-based file format, which can be downloaded and used locally. To allow fast access, the catalogs are partitioned into hierarchical triangular meshes and stored in HDF5 files. Several tools are provided to perform efficient cone searches at resolutions spanning from a few arc seconds to degrees, within a few milliseconds time. The first released version includes the following catalogs (by alphabetical order): 2MASS, 2MASS ext…
A novel method to infer logical relationships between sets is presented. These sets can be any collection of elements, for example astronomical catalogs of celestial objects. The method does not require the contents of the sets to be known explicitly. It combines incomplete knowledge about the relationships between sets to infer a priori unknown relationships. Relationships between sets are represented by sets of Boolean hypercubes. This leads to deductive reasoning by application of logical operators to these sets of hypercubes. A pseudocode for an efficient implementation is described. The m…
Multi-wavelength astronomical studies require cross-identification of detections of the same celestial objects in multiple catalogs based on spherical coordinates and other properties. Because of the large data volumes and spherical geometry, the symmetric N-way association of astronomical detections is a computationally intensive problem, even when sophisticated indexing schemes are used to exclude obviously false candidates. Legacy astronomical catalogs already contain detections of more than a hundred million objects while the ongoing and future surveys will produce catalogs of billions of …
Cross-matching operation, which is to find corresponding data for the same celestial object or region from multiple catalogues,is indispensable to astronomical data analysis and research. Due to the large amount of astronomical catalogues generated by the ongoing and next-generation large-scale sky surveys, the time complexity of the cross-matching is increasing dramatically. Heterogeneous computing environments provide a theoretical possibility to accelerate the cross-matching, but the performance advantages of heterogeneous computing resources have not been fully utilized. To meet the challe…
University of California – Berkeley（加州大学伯克利分校）；Columbia University（哥伦比亚大学）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）；Princeton University（普林斯顿大学）；The Voleon Group（Voleon集团）
We present a new, fully generative model for constructing astronomical catalogs from optical telescope image sets. Each pixel intensity is treated as a random variable with parameters that depend on the latent properties of stars and galaxies. These latent properties are themselves modeled as random. We compare two procedures for posterior inference. One procedure is based on Markov chain Monte Carlo (MCMC) while the other is based on variational inference (VI). The MCMC procedure excels at quantifying uncertainty, while the VI procedure is 1000 times faster. On a supercomputer, the VI procedu…
University of California, Berkeley（加州大学伯克利分校）；Intel Corporation（英特尔公司）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）
Celeste is a procedure for inferring astronomical catalogs that attains state-of-the-art scientific results. To date, Celeste has been scaled to at most hundreds of megabytes of astronomical images: Bayesian posterior inference is notoriously demanding computationally. In this paper, we report on a scalable, parallel version of Celeste, suitable for learning catalogs from modern large-scale astronomical datasets. Our algorithmic innovations include a fast numerical optimization routine for Bayesian posterior inference and a statistically efficient scheme for decomposing astronomical optimizati…
Scientific exploitation of the ever increasing volumes of astronomical data requires efficient and practical methods for data access, visualisation, and analysis. Hierarchical sky tessellation techniques enable a multi-resolution approach to organising data on angular scales from the full sky down to the individual image pixels. Aims. We aim to show that the Hierarchical progressive survey (HiPS) scheme for describing astronomical images, source catalogues, and three-dimensional data cubes is a practical solution to managing large volumes of heterogeneous data and that it enables a new level o…
Time series data of celestial objects are commonly used to study valuable and unexpected objects such as extrasolar planets and supernova in time domain astronomy. Due to the rapid growth of data volume, traditional manual methods are becoming extremely hard and infeasible for continuously analyzing accumulated observation data. To meet such demands, we designed and implemented a special tool named AstroCatR that can efficiently and flexibly reconstruct time series data from large-scale astronomical catalogues. AstroCatR can load original catalogue data from Flexible Image Transport System (FI…
Universidad De Concepcion（康塞普西翁大学）；Millennium Institute of Astrophysics（千年天体物理研究所）；Harvard University（哈佛大学）；University College London（伦敦大学学院）；Universidad de Chile（智利大学）
In the new era of very large telescopes, where data is crucial to expand scientific knowledge, we have witnessed many deep learning applications for the automatic classification of lightcurves. Recurrent neural networks (RNNs) are one of the models used for these applications, and the LSTM unit stands out for being an excellent choice for the representation of long time series. In general, RNNs assume observations at discrete times, which may not suit the irregular sampling of lightcurves. A traditional technique to address irregular sequences consists of adding the sampling time to the networ…
Neural Posterior Estimation for Cataloging Astronomical Images with Spatially Varying Backgrounds and Point Spread Functions
Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI 48109, USA（电气工程与计算机科学系，密歇根大学，安娜堡分校）；Department of Physics and Astronomy and PITT PACC, University of Pittsburgh, Pittsburgh, PA 15260, USA（物理与天文学系及PITT PACC，匹兹堡大学，匹兹堡分校）；Department of Physics, University of Michigan, Ann Arbor, MI 48109, USA（物理系，密歇根大学，安娜堡分校）；Leinweber Center for Theoretical Physics, University of Michigan, Ann Arbor, MI 48109, USA（理论物理中心，密歇根大学，安娜堡分校）；Department of Statistics, University of Michigan, Ann Arbor, MI 48109, USA（统计学系，密歇根大学，安娜堡分校）
We present a new probabilistic method for detecting, deblending, and cataloging astronomical sources called the Bayesian Light Source Separator (BLISS). BLISS is based on deep generative models, which embed neural networks within a Bayesian model. For posterior inference, BLISS uses a new form of variational inference known as Forward Amortized Variational Inference. The BLISS inference routine is fast, requiring a single forward pass of the encoder networks on a GPU once the encoder networks are trained. BLISS can perform fully Bayesian inference on megapixel images in seconds, and produces h…
The Legacy Survey of Space and Time, operated by the Vera C. Rubin Observatory, is a 10-year astronomical survey due to start operations in 2022 that will image half the sky every three nights. LSST will produce ~20TB of raw data per night which will be calibrated and analyzed in almost real time. Given the volume of LSST data, the traditional subset-download-process paradigm of data reprocessing faces significant challenges. We describe here, the first steps towards a gateway for astronomical science that would enable astronomers to analyze images and catalogs at scale. In this first step we …
The availability of vast quantities of data through electronic archives has transformed astronomical research. It has also enabled the creation of new products, models and simulations, often from distributed input data and models, that are themselves made electronically available. These products will only provide maximal long-term value to astronomers when accompanied by records of their provenance; that is, records of the data and processes used in the creation of such products. We use the creation of image mosaics with the Montage grid-enabled mosaic engine to emphasize the necessity of prov…
The ever-growing datasets in observational astronomy have challenged scientists in many aspects, including an efficient and interactive data exploration and visualization. Many tools have been developed to confront this challenge. However, they usually focus on displaying the actual images or focus on visualizing patterns within catalogs in a predefined way. In this paper we introduce Vizic, a Python visualization library that builds the connection between images and catalogs through an interactive map of the sky region. Vizic visualizes catalog data over a custom background canvas using the s…
We have built a reliable and robust system that takes as input an astronomical image, and returns as output the pointing, scale, and orientation of that image (the astrometric calibration or WCS information). The system requires no first guess, and works with the information in the image pixels alone; that is, the problem is a generalization of the "lost in space" problem in which nothing--not even the image scale--is known. After robust source detection is performed in the input image, asterisms (sets of four or five stars) are geometrically hashed and compared to pre-indexed hashes to genera…
I present a software tool for solving the astrometry of astronomical images. The code puts emphasis on robustness against failures for correctly matching the sources in the image to a reference catalog, and on the stability of the solutions over the field of view (e.g., using orthogonal polynomials for the fitted transformation). The code was tested on over 50,000 images from various sources, including the Palomar Transient Factory (PTF) and the Zwicky Transient Facility (ZTF). The tested images equally represent low and high Galactic latitude fields and exhibit failure/bad-solution rate of <2…
This specification defines a simple query protocol for retrieving records from a catalog of astronomical sources. The query describes sky position and an angular distance, defining a cone on the sky. The response returns a list of astronomical sources from the catalog whose positions lie within the cone, formatted as a VOTable. This version of the specification is essentially a transcription of the original Cone Search specification in order to move it into the IVOA standardization process.
We introduce AXS (Astronomy eXtensions for Spark), a scalable open-source astronomical data analysis framework built on Apache Spark, a widely used industry-standard engine for big data processing. Building on capabilities present in Spark, AXS aims to enable querying and analyzing almost arbitrarily large astronomical catalogs using familiar Python/AstroPy concepts, DataFrame APIs, and SQL statements. We achieve this by i) adding support to Spark for efficient on-line positional cross-matching and ii) supplying a Python library supporting commonly-used operations for astronomical data analysi…
We present a Bayesian method for the identification and classification of objects from sets of astronomical catalogs, given a predefined classification scheme. Identification refers here to the association of entries in different catalogs to a single object, and classification refers to the matching of the associated data set to a model selected from a set of parametrized models of different complexity. By the virtue of Bayes' theorem, we can combine both tasks in an efficient way, which allows a largely automated and still reliable way to generate classified astronomical catalogs. A problem t…
ESASky is a science-driven discovery portal to explore the multi-wavelength sky and visualise and access multiple astronomical archive holdings. The tool is a web application that requires no prior knowledge of any of the missions involved and gives users world-wide simplified access to the highest-level science data products from multiple astronomical space-based astronomy missions plus a number of ESA source catalogues. The first public release of ESASky features interfaces for the visualisation of the sky in multiple wavelengths, the visualisation of query results summaries, and the visuali…
This elementary review covers the basics of working with astronomical data, notably with images, spectra and higher-level (catalog) data. The basic concepts and tools are presented using both application software (DS9 and TOPCAT) and Python. The level of presentation is suitable for undergraduate students, but should also be accessible to advanced high school students.
A High-Throughput AES-GCM Implementation on GPUs for Secure, Policy-Based Access to Massive Astronomical Catalogs
Light Bridges S.L.（Light Bridges 有限公司）；Observatorio Astronómico del Teide（泰德天文台）；Universidad de La Laguna (ULL)（拉古纳大学）；Instituto Tecnológico y de Energías Renovables (ITER)（技术与可再生能源研究所）；Instituto de Astrofísica de Canarias (IAC)（加那利天体物理研究所）
The Millennium Institute of Astrophysics（千年天体物理研究所）；Pontificia Universidad Católica de Chile（智利天主教大学）；Harvard University（哈佛大学）；Harvard-Smithsonian Center for Astrophysics（哈佛-史密森天体物理中心）；Max-Planck Institute for Astronomy（马克斯·普朗克天文研究所）
The development of synoptic sky surveys has led to a massive amount of data for which resources needed for analysis are beyond human capabilities. To process this information and to extract all possible knowledge, machine learning techniques become necessary. Here we present a new method to automatically discover unknown variable objects in large astronomical catalogs. With the aim of taking full advantage of all the information we have about known objects, our method is based on a supervised algorithm. In particular, we train a random forest classifier using known variability classes of objec…
VST--Tube is a new software package designed to process optical astronomical images. It is an automated pipeline to go from the raw exposures to fully calibrated co-added images, and to extract catalogs with aperture and PSF photometry. A set of tools allow the data administration and the quality check of the intermediate and final products. VST-Tube comes with a Graphical User Interface to facilitate the interaction between data and user. We outline here the VST--Tube architecture and show some applications enlightening some of the characteristics of the pipeline.
Astronomical catalogs derived from wide-field imaging surveys are an important tool for understanding the Universe. We construct an astronomical catalog from 55 TB of imaging data using Celeste, a Bayesian variational inference code written entirely in the high-productivity programming language Julia. Using over 1.3 million threads on 650,000 Intel Xeon Phi cores of the Cori Phase II supercomputer, Celeste achieves a peak rate of 1.54 DP PFLOP/s. Celeste is able to jointly optimize parameters for 188M stars and galaxies, loading and processing 178 TB across 8192 nodes in 14.6 minutes. To achie…
Matching astronomical catalogs in crowded regions of the sky is challenging both statistically and computationally due to the many possible alternative associations. Budavári and Basu (2016) modeled the two-catalog situation as an Assignment Problem and used the famous Hungarian algorithm to solve it. Here we treat cross-identification of multiple catalogs by introducing a different approach based on integer linear programming. We first test this new method on problems with two catalogs and compare with the previous results. We then test the efficacy of the new approach on problems with three …
Recent all-sky and large-area astronomical surveys and their catalogued data over the whole range of electromagnetic spectrum are reviewed, from Gamma-ray to radio, such as Fermi-GLAST and INTEGRAL in Gamma-ray, ROSAT, XMM and Chandra in X-ray, GALEX in UV, SDSS and several POSS I and II based catalogues (APM, MAPS, USNO, GSC) in optical range, 2MASS in NIR, WISE and AKARI IRC in MIR, IRAS and AKARI FIS in FIR, NVSS and FIRST in radio and many others, as well as most important surveys giving optical images (DSS I and II, SDSS, etc.), proper motions (Tycho, USNO, Gaia), variability (GCVS, NSVS,…
The first step in a science project is the acquisition and understanding of the relevant data. This paper outlines the results of a project to design and test network tools specifically oriented at retrieving astronomical data. The tools range from simple data transfer methods to more complex browser-emulating scripts. When integrated with a defined sample or catalog, these scripts provide seamless techniques to retrieve and store data of varying types. Examples are given on how these tools can be used to leapfrog from website to website to acquire multi-wavelength datasets. This project demon…
The next generation of telescopes will acquire terabytes of image data on a nightly basis. Collectively, these large images will contain billions of interesting objects, which astronomers call sources. The astronomers' task is to construct a catalog detailing the coordinates and other properties of the sources. The source catalog is the primary data product for most telescopes and is an important input for testing new astrophysical theories, but to construct the catalog one must first detect the sources. Existing algorithms for catalog creation are effective at detecting sources, but do not ha…
BGRem: A background noise remover for astronomical images based on a diffusion model
Radboud University Nijmegen（奈梅亨拉德堡德大学）；Nikhef（荷兰国家核子与粒子物理研究所）；University of Oxford（牛津大学）；University of Nova Gorica（新戈里察大学）；Instituto de Física Corpuscular (IFIC), CSIC-UV（瓦伦西亚-西班牙国家科学研究委员会粒子物理研究所）；University of Cape Town（开普敦大学）；South African Astronomical Observatory（南非天文台）
The completeness of source detection is critical for analyzing the photometric and spatial properties of the population of interest observed by astronomical imaging. We present a software package $\mathtt{ComEst}$, which calculates the completeness of source detection on charge-coupled device (CCD) images of astronomical observations, especially for the optical and near-infrared (NIR) imaging of galaxies and point sources. The completeness estimator $\mathtt{ComEst}$ is designed for the source finder $\mathtt{SExtractor}$ used on the CCD images saved in the Flexible Image Transport System (FIT…
We present a classification-based approach to identify quasi-stellar radio sources (quasars) in the Sloan Digital Sky Survey and evaluate its performance on a manually labeled training set. While reasonable results can already be obtained via approaches working only on photometric data, our experiments indicate that simple but problem-specific features extracted from spectroscopic data can significantly improve the classification performance. Since our approach works orthogonal to existing classification schemes used for building the spectroscopic catalogs, our classification results are well …
The days of the lone astronomer with his optical telescope and photographic plates are long gone: Astronomy in 2025 will not only be multi-wavelength, but multi-messenger, and dominated by huge data sets and matching data rates. Catalogues listing detailed properties of billions of objects will in themselves require a new industrial-scale approach to scientific discovery, requiring the latest techniques of advanced data analytics and an early engagement with the first generation of cognitive computing systems. Astronomers have the opportunity to be early adopters of these new technologies and …
Powellsnakes is a Bayesian algorithm for detecting compact objects embedded in a diffuse background, and was selected and successfully employed by the Planck consortium in the production of its first public deliverable: the Early Release Compact Source Catalogue (ERCSC). We present the critical foundations and main directions of further development of PwS, which extend it in terms of formal correctness and the optimal use of all the available information in a consistent unified framework, where no distinction is made between point sources (unresolved objects), SZ clusters, single or multi-chan…
Founded in 2010, the Taiwan Extragalactic Astronomical Data Center (TWEA-DC) has for goal to propose access to large amount of data for the Taiwanese and International community, focusing its efforts on Extragalactic science. In continuation with individual efforts in Taiwan over the past few years, this is the first steppingstone towards the building of a National Virtual Observatory. Taking advantage of our own fast indexing algorithm (BLINK), based on a octahedral meshing of the sky coupled with a very fast kd-tree and a clever parallelization amongst available resources, TWEA-DC will propo…
The emerging need for efficient, reliable and scalable astronomical catalog cross-matching is becoming more pressing in the current data-driven science era, where the size of data has rapidly increased up to the Petabyte scale. C3 (Command-line Catalogue Cross-matching) is a multi-platform tool designed to efficiently cross-match massive catalogues from modern astronomical surveys, ensuring high-performance capabilities through the use of a multi-core parallel processing paradigm. The tool has been conceived to be executed as a stand-alone command-line process or integrated within any generic …
Astronomical data take on a multitude of forms -- catalogs, data cubes, images, and simulations. The availability of software for rendering high-quality three-dimensional graphics lends itself to the paradigm of exploring the incredible parameter space afforded by the astronomical sciences. The software program Blender gives astronomers a useful tool for displaying data in a manner used by three-dimensional (3D) graphics specialists and animators. The interface to this popular software package is introduced with attention to features of interest in astronomy. An overview of the steps for gener…
We present a novel approach to quality control during the processing of astronomical data. Quality control in the Astro-WISE Information System is integral to all aspects of data handing and provides transparent access to quality estimators for all stages of data reduction from the raw image to the final catalog. The implementation of quality control mechanisms relies on the core features in this Astro-WISE Environment (AWE): an object-oriented framework, full data lineage, and both forward and backward chaining. Quality control information can be accessed via the command-line awe-prompt and t…
We report the results of a study to compare the performance of two common database indexing methods, HTM and HEALPix, on Solaris and Windows database servers installed with PostgreSQL, and a Windows Server installed with MS SQL Server. The indexing was applied to the 2MASS All-Sky Catalog and to the Hubble Source Catalog, which approximate the diversity of catalogs common in astronomy. On each server, the study compared indexing performance by submitting 1 million queries at each index level with random sky positions and random cone search radius, which was computed on a logarithmic scale betw…
Statistical studies of astronomical data sets, in particular of cataloged properties for discrete objects, are central to astrophysics. One cannot model those objects' population properties or incidences without a quantitative understanding of the conditions under which these objects ended up in a catalog or sample, the sample's selection function. As systematic and didactic introductions to this topic are scarce in the astrophysical literature, we aim to provide one, addressing generically the following questions: What is a selection function? What arguments $\vec{q}$ should a selection funct…
This tutorial is devoted to extending an existing catalogue with data taken elsewhere, either from CDS Vizier or Simbad database. As an example, we used the so-called 'Spectroscopic Survey of Stars in the Solar Neighborhood' (aka. S4N, Allende Prieto et al. 2004) in order to retrieve all objects with available data for the set of fundamental stellar parameters effective temperature, surface gravity and metallicity. Then for each object in this dataset we query Simbad database to retrieve the projected rotational velocity. This combines Vizier and Simbad queries made using Python astroquery mod…
We present AstroVaDEr, a variational autoencoder designed to perform unsupervised clustering and synthetic image generation using astronomical imaging catalogues. The model is a convolutional neural network that learns to embed images into a low dimensional latent space, and simultaneously optimises a Gaussian Mixture Model (GMM) on the embedded vectors to cluster the training data. By utilising variational inference, we are able to use the learned GMM as a statistical prior on the latent space to facilitate random sampling and generation of synthetic images. We demonstrate AstroVaDEr's capabi…
The increasing number of Very High Energy (VHE) sources discovered by the current generation of Cherenkov telescopes made particularly relevant the creation of a dedicated source catalogs as well as the cross-correlation of VHE and lower energy bands data in a multi-wavelength framework. The "TeGeV Catalog" hosted at the ASI Science Data Center (ASDC) is a catalog of VHE sources detected by ground-based Cherenkov detectors. The TeGeVcat collects all the relevant information publicly available about the observed GeV/TeV sources. The catalog contains also information about public light curves wh…
The table access protocol (TAP) defines a service protocol for accessing general table data, including astronomical catalogs as well as general database tables. Access is provided for both database and table metadata as well as for actual table data. This version of the protocol includes support for multiple query languages, including queries specified using the Astronomical Data Query Language (ADQL [1]) and the Parameterised Query Language (PQL, under development) within an integrated interface. It also includes support for both synchronous and asynchronous queries. Special support is provid…
Astronomical Refutation of the New Chronology by Fomenko and Nosovsky: The 1151-Year Planetary Cycle and Dating of the Almagest via Speed/Error Correlation
