Cross-matching operation, which is to find corresponding data for the same celestial object or region from multiple catalogues,is indispensable to astronomical data analysis and research. Due to the large amount of astronomical catalogues generated by the ongoing and next-generation large-scale sky surveys, the time complexity of the cross-matching is increasing dramatically. Heterogeneous computing environments provide a theoretical possibility to accelerate the cross-matching, but the performance advantages of heterogeneous computing resources have not been fully utilized. To meet the challenge of cross-matching for substantial increasing amount of astronomical observation data, this paper proposes Heterogeneous-computing-enabled Large Catalogue Cross-matcher (HLC2), a high-performance cross-matching framework based on spherical position deviation on CPU-GPU heterogeneous computing platforms. It supports scalable and flexible cross-matching and can be directly applied to the fusion of large astronomical cataloguesfrom survey missions and astronomical data centres. A performance estimation model is proposed to locate the performance bottlenecks and guide the optimizations. A two-level partitioning strategy is designed to generate an optimized data placement according to the positions of celestial objects to increase throughput. To make HLC2 a more adaptive solution, the architecture-aware task splitting, thread parallelization, and concurrent scheduling strategies are designed and integrated. Moreover, a novel quad-direction strategy is proposed for the boundary problem to effectively balance performance and completeness. We have experimentally evaluated HLC2 using public released catalogue data. Experiments demonstrate that HLC2 scales well on different sizes of catalogues and the cross-matching speed is significantly improved compared to the state-of-the-art cross-matchers.
In this paper we presented the algorithm designed to efficient coordinate cross-match of objects in the modern massive astronomical catalogues. Preliminary data sort in the existed catalogues provides the opportunity for coordinate identification of the objects without any constraints with the storage and technical environment (PC). Using the multi-threading of the modern computing processors allows speeding up the program up to read-write data to the storage. Also the paper contains the main difficulties of implementing of the algorithm, as well as their possible solutions.
I present a new kind of astronomical database based on small text files and a distributed version control system. This encourages the community to work collaboratively. It creates a decentralized, completely open and democratic way of managing small to medium sized heterogeneous astronomical databases and catalogues. The use of the XML file format allows an easy to parse and read, yet dynamic and extendable database structure. The Open Exoplanet Catalogue is based on these principles and presented as an example. It is a catalogue of all discovered extra-solar planets. It is the only catalogue …
Scientific exploitation of the ever increasing volumes of astronomical data requires efficient and practical methods for data access, visualisation, and analysis. Hierarchical sky tessellation techniques enable a multi-resolution approach to organising data on angular scales from the full sky down to the individual image pixels. Aims. We aim to show that the Hierarchical progressive survey (HiPS) scheme for describing astronomical images, source catalogues, and three-dimensional data cubes is a practical solution to managing large volumes of heterogeneous data and that it enables a new level o…
Time series data of celestial objects are commonly used to study valuable and unexpected objects such as extrasolar planets and supernova in time domain astronomy. Due to the rapid growth of data volume, traditional manual methods are becoming extremely hard and infeasible for continuously analyzing accumulated observation data. To meet such demands, we designed and implemented a special tool named AstroCatR that can efficiently and flexibly reconstruct time series data from large-scale astronomical catalogues. AstroCatR can load original catalogue data from Flexible Image Transport System (FI…
Large catalogues are ubiquitous throughout astronomy, but most scientific analyses are carried out on smaller samples selected from these catalogues by chosen cuts on catalogued quantities. The selection function of that scientific sample - the probability that a star in the catalogue will satisfy these cuts and so make it into the sample - is thus unique to each scientific analysis. We have created a general framework that can flexibly estimate the selection function of a sample drawn from a catalogue in terms of position, magnitude and colour. Our method is unique in using the binomial likel…
Astronomical instruments make intensity measurements; any precise astronomical experiment ought to involve modeling those measurements. People make catalogues, but because a catalogue requires hard decisions about calibration and detection, no catalogue can contain all of the information in the raw pixels relevant to most scientific investigations. Here we advocate making catalogue-like data outputs that permit investigators to test hypotheses with almost the power of the original image pixels. The key is to provide users with approximations to likelihood tests against the raw image pixels. We…
ESASky is a science-driven discovery portal to explore the multi-wavelength sky and visualise and access multiple astronomical archive holdings. The tool is a web application that requires no prior knowledge of any of the missions involved and gives users world-wide simplified access to the highest-level science data products from multiple astronomical space-based astronomy missions plus a number of ESA source catalogues. The first public release of ESASky features interfaces for the visualisation of the sky in multiple wavelengths, the visualisation of query results summaries, and the visuali…
Recent all-sky and large-area astronomical surveys and their catalogued data over the whole range of electromagnetic spectrum are reviewed, from Gamma-ray to radio, such as Fermi-GLAST and INTEGRAL in Gamma-ray, ROSAT, XMM and Chandra in X-ray, GALEX in UV, SDSS and several POSS I and II based catalogues (APM, MAPS, USNO, GSC) in optical range, 2MASS in NIR, WISE and AKARI IRC in MIR, IRAS and AKARI FIS in FIR, NVSS and FIRST in radio and many others, as well as most important surveys giving optical images (DSS I and II, SDSS, etc.), proper motions (Tycho, USNO, Gaia), variability (GCVS, NSVS,…
Although the Gaia catalogue on its own will be a very powerful tool, it is the combination of this highly accurate archive with other archives that will truly open up amazing possibilities for astronomical research. The advanced interoperation of archives is based on cross-matching, leaving the user with the feeling of working with one single data archive. The data retrieval should work not only across data archives, but also across wavelength domains. The first step for seamless data access is the computation of the cross-match between Gaia and external surveys. The matching of astronomical c…
In the current data-driven science era, it is needed that data analysis techniques has to quickly evolve to face with data whose dimensions has increased up to the Petabyte scale. In particular, being modern astrophysics based on multi-wavelength data organized into large catalogues, it is crucial that the astronomical catalog cross-matching methods, strongly dependant from the catalogues size, must ensure efficiency, reliability and scalability. Furthermore, multi-band data are archived and reduced in different ways, so that the resulting catalogues may differ each other in formats, resolutio…
The days of the lone astronomer with his optical telescope and photographic plates are long gone: Astronomy in 2025 will not only be multi-wavelength, but multi-messenger, and dominated by huge data sets and matching data rates. Catalogues listing detailed properties of billions of objects will in themselves require a new industrial-scale approach to scientific discovery, requiring the latest techniques of advanced data analytics and an early engagement with the first generation of cognitive computing systems. Astronomers have the opportunity to be early adopters of these new technologies and …
Powellsnakes is a Bayesian algorithm for detecting compact objects embedded in a diffuse background, and was selected and successfully employed by the Planck consortium in the production of its first public deliverable: the Early Release Compact Source Catalogue (ERCSC). We present the critical foundations and main directions of further development of PwS, which extend it in terms of formal correctness and the optimal use of all the available information in a consistent unified framework, where no distinction is made between point sources (unresolved objects), SZ clusters, single or multi-chan…
Modern Astrophysics is based on multi-wavelength data organized into large and heterogeneous catalogues. Hence, the need for efficient, reliable and scalable catalogue cross-matching methods plays a crucial role in the era of the petabyte scale. Furthermore, multi-band data have often very different angular resolution, requiring the highest generality of cross-matching features, mainly in terms of region shape and resolution. In this work we present $C^{3}$ (Command-line Catalogue Cross-match), a multi-platform application designed to efficiently cross-match massive catalogues. It is based on …
This tutorial is devoted to extending an existing catalogue with data taken elsewhere, either from CDS Vizier or Simbad database. As an example, we used the so-called 'Spectroscopic Survey of Stars in the Solar Neighborhood' (aka. S4N, Allende Prieto et al. 2004) in order to retrieve all objects with available data for the set of fundamental stellar parameters effective temperature, surface gravity and metallicity. Then for each object in this dataset we query Simbad database to retrieve the projected rotational velocity. This combines Vizier and Simbad queries made using Python astroquery mod…
The Surviving Fragment of the Zij-i Malikshahi: A Preliminary Edition, Translation, and Study of the 100-Star Catalogue in BnF Arabe 5968
S. U. Umarov Physical-Technical Institute, National Academy of Sciences of Tajikistan（塔吉克斯坦科学院 S.U. 乌马罗夫物理技术研究所）；Institute of Astrophysics, National Academy of Sciences of Tajikistan（塔吉克斯坦科学院天体物理研究所）
Context. Although the Gaia catalogue on its own is a very powerful tool, it is the combination of this high-accuracy archive with other archives that will truly open up amazing possibilities for astronomical research. The advanced interoperation of archives is based on cross-matching, leaving the user with the feeling of working with one single data archive. The data retrieval should work not only across data archives but also across wavelength domains. The first step for a seamless access to the data is the computation of the cross-match between Gaia and external surveys. Aims. We describe th…
ARCHES (Astronomical Resource Cross-matching for High Energy Studies) is a FP7-Space funded project whose aim is to provide the international astronomical community with well-characterised multi-wavelength data in the form of spectral energy distributions (SEDs) for large samples of objects extracted from the 3XMM DR5 X-ray catalogue of serendipitous sources. The project has developed new tools implementing fully probabilistic simultaneous cross-correlation of several catalogues for unresolved sources and a multi-wavelength finder for clusters of galaxies for extended sources. These enhanced r…
The emerging need for efficient, reliable and scalable astronomical catalog cross-matching is becoming more pressing in the current data-driven science era, where the size of data has rapidly increased up to the Petabyte scale. C3 (Command-line Catalogue Cross-matching) is a multi-platform tool designed to efficiently cross-match massive catalogues from modern astronomical surveys, ensuring high-performance capabilities through the use of a multi-core parallel processing paradigm. The tool has been conceived to be executed as a stand-alone command-line process or integrated within any generic …
The time domain has been identified as one of the most important areas of astronomical research for the next decade. The Virtual Observatory is in the vanguard with dedicated tools and services that enable and facilitate the discovery, dissemination and analysis of time domain data. These range in scope from rapid notifications of time-critical astronomical transients to annotating long-term variables with the latest modeling results. In this paper, we will review the prior art in these areas and focus on the capabilities that the VAO is bringing to bear in support of time domain science. In p…
Astronomers are good at sharing data, but poorer at sharing knowledge. Almost all astronomical data ends up in open archives, and access to these is being simplified by the development of the global Virtual Observatory (VO). This is a great advance, but the fundamental problem remains that these archives contain only basic observational data, whereas all the astrophysical interpretation of that data -- which source is a quasar, which a low-mass star, and which an image artefact -- is contained in journal papers, with very little linkage back from the literature to the original data archives. I…
We lay the foundations of a statistical framework for multi-catalogue cross-correlation and cross-identification based on explicit simplified catalogue models. A proper identification process should rely on both astrometric and photometric data. Under some conditions, the astrometric part and the photometric part can be processed separately and merged a posteriori to provide a single global probability of identification. The present paper addresses almost exclusively the astrometrical part and specifies the proper probabilities to be merged with photometric likelihoods. To select matching cand…
We present an application of unsupervised machine learning - the self-organised map (SOM) - as a tool for visualising, exploring and mining the catalogues of large astronomical surveys. Self-organisation culminates in a low-resolution representation of the 'topology' of a parameter volume, and this can be exploited in various ways pertinent to astronomy. Using data from the Cosmological Evolution Survey (COSMOS), we demonstrate two key astronomical applications of the SOM: (i) object classification and selection, using the example of galaxies with active galactic nuclei as a demonstration, and…
As research datasets and analyses grow in complexity, data that could be valuable to other researchers and to support the integrity of published work remain uncurated across disciplines. These data are especially concentrated in the Long Tail of funded research, where curation resources and related expertise are often inaccessible. In the domain of astronomy, it is undisputed that uncurated dark data exist, but the scope of the problem remains uncertain. The Astrolabe Project is a collaboration between University of Arizona researchers, the CyVerse cyberinfrastructure environment, and American…
This document defines the high level metadata necessary to describe the physical parameter space of observed or simulated astronomical data sets, such as 2D-images, data cubes, X-ray event lists, IFU data, etc.. The Characterisation data model is an abstraction which can be used to derive a structured description of any relevant data and thus to facilitate its discovery and scientific interpretation. The model aims at facilitating the manipulation of heterogeneous data in any VO framework or portal. A VO Characterisation instance can include descriptions of the data axes, the range of coordina…
The Hong Kong/AAO/Strasbourg Halpha (HASH) planetary nebula database is an online research platform providing free and easy access to the largest and most comprehensive catalogue of known Galactic planetary nebulae (PNe) and a repository of observational data (imaging and spectroscopy) for these and related astronomical objects. The main motivation for creating this system is resolving some of long standing problems in the field e.g. problems with mimics and dubious and/or misidentifications, errors in observational data and consolidation of the widely scattered data-sets. This facility allows…
We present CosmoHub (https://cosmohub.pic.es), a web application based on Hadoop to perform interactive exploration and distribution of massive cosmological datasets. Recent Cosmology seeks to unveil the nature of both dark matter and dark energy mapping the large-scale structure of the Universe, through the analysis of massive amounts of astronomical data, progressively increasing during the last (and future) decades with the digitization and automation of the experimental techniques. CosmoHub, hosted and developed at the Port d'Informació Científica (PIC), provides support to a worldwide com…
ESA Gaia mission is producing the more accurate source catalogue in astronomy up to now. That represents a challenge on the archiving area to make accessible this information to the astronomers in an efficient way. Also, new astronomical missions have reinforced the change on the development of archives. Archives, as simple applications to access the data are being evolving into complex data center structures where computing power services are available for users and data mining tools are integrated into the server side. In the case of astronomy science that involves the use of big catalogues,…
We present AstroVaDEr, a variational autoencoder designed to perform unsupervised clustering and synthetic image generation using astronomical imaging catalogues. The model is a convolutional neural network that learns to embed images into a low dimensional latent space, and simultaneously optimises a Gaussian Mixture Model (GMM) on the embedded vectors to cluster the training data. By utilising variational inference, we are able to use the learned GMM as a statistical prior on the latent space to facilitate random sampling and generation of synthetic images. We demonstrate AstroVaDEr's capabi…
An automated, rapid classification of transient events detected in the modern synoptic sky surveys is essential for their scientific utility and effective follow-up using scarce resources. This presents some unusual challenges: the data are sparse, heterogeneous and incomplete; evolving in time; and most of the relevant information comes not from the data stream itself, but from a variety of archival data and contextual information (spatial, temporal, and multi-wavelength). We are exploring a variety of novel techniques, mostly Bayesian, to respond to these challenges, using the ongoing CRTS s…
Operation of the US Virtual Astronomical Observatory shares some issues with modern physical observatories, e.g., intimidating data volumes and rapid technological change, and must also address unique concerns like the lack of direct control of the underlying and scattered data resources, and the distributed nature of the observatory itself. In this paper we discuss how the VAO has addressed these challenges to provide the astronomical community with a coherent set of science-enabling tools and services. The distributed nature of our virtual observatory-with data and personnel spanning geograp…
The NEMESIS general YSO catalogue - I. Supervised classification with deep learning methods
Konkoly Observatory, Research Centre for Astronomy and Earth Sciences, Hungarian Research Network (HUN-REN)（孔科利天文台，地球科学研究中心，匈牙利研究网络）；CSFK, MTA Centre of Excellence（CSFK，马塔卓越中心）；Department of Experimental Physics, Institute of Physics, University of Szeged（塞格德大学物理学院实验物理系）；Université de Genève, Department of Astronomy（日内瓦大学天文学系）；Université de Genève, Department of Basic Neuroscience（日内瓦大学基础神经科学系）；Universität Wien, Institut für Astrophysik（维也纳大学天体物理学研究所）
The Rapid ASKAP Continuum Survey (RACS) VIII: total intensity and circular polarization images and catalogues at 887.5 MHz from RACS-low2
CSIRO Space and Astronomy（CSIRO空间与天文学）；Sydney Institute for Astronomy, School of Physics, The University of Sydney（悉尼大学物理学院悉尼天文研究所）；Australian SKA Regional Centre (AusSRC), Curtin University（科廷大学澳大利亚SKA区域中心）；SKA Observatory, SKA-Low Science Operations Centre（SKA天文台SKA低频科学运营中心）
Astronomical data are gathered through a very large number of heterogeneous techniques and stored in very diversified and often incompatible data repositories. Moreover in the e-science environment, it is needed to integrate services across distributed, heterogeneous, dynamic "virtual organizations" formed by different resources within a single enterprise and/or external resource sharing and service provider relationships. The DAME/VONeural project, run jointly by the University Federico II, INAF (National Institute of Astrophysics) Astronomical Observatories of Napoli and the California Insti…
The Pennsylvania State University（宾夕法尼亚州立大学）；University of Toronto（多伦多大学）
Upcoming astronomical surveys will observe billions of galaxies across cosmic time, providing a unique opportunity to map the many pathways of galaxy assembly to an incredibly high resolution. However, the huge amount of data also poses an immediate computational challenge: current tools for inferring parameters from the light of galaxies take $\gtrsim 10$ hours per fit. This is prohibitively expensive. Simulation-based Inference (SBI) is a promising solution. However, it requires simulated data with identical characteristics to the observed data, whereas real astronomical surveys are often hi…
