Throughout the processing and analysis of survey data, a ubiquitous issue nowadays is that we are spoilt for choice when we need to select a methodology for some of its steps. The alternative methods usually fail and excel in different data regions, and have various advantages and drawbacks, so a combination that unites the strengths of all while suppressing the weaknesses is desirable. We propose to use a two-level hierarchy of learners. Its first level consists of training and applying the possible base methods on the first part of a known set. At the second level, we feed the output probability distributions from all base methods to a second learner trained on the remaining known objects. Using classification of variable stars and photometric redshift estimation as examples, we show that the hierarchical combination is capable of achieving general improvement over averaging-type combination methods, correcting systematics present in all base methods, is easy to train and apply, and thus, it is a promising tool in the astronomical "Big Data" era.
This is the 3rd version with major updates and revisions to the 2nd version of the same title. In this article we have collected and corrected some common mistakes made by Chinese students in writing astronomy and physics literature in English. Brief explanations of these mistakes are given in Chinese. We plan to continue to update this collection periodically. Comments, suggestions and criticisms are welcome.
Hierarchical clustering is a common algorithm in data analysis. It is unique among many clustering algorithms in that it draws dendrograms based on the distance of data under a certain metric, and group them. It is widely used in all areas of astronomical research, covering various scales from asteroids and molecular clouds, to galaxies and galaxy cluster. This paper systematically reviews the history and current status of the development of hierarchical clustering methods in various branches of astronomy. These applications can be grouped into two broad categories, one revealing the intrinsic…
We present a review of data types and statistical methods often encountered in astronomy. The aim is to provide an introduction to statistical applications in astronomy for statisticians and computer scientists. We highlight the complex, often hierarchical, nature of many astronomy inference problems and advocate for cross-disciplinary collaborations to address these challenges.
HEALPix -- the Hierarchical Equal Area isoLatitude Pixelization -- has become a standard in high-energy and gravitational wave astronomy. Originally developed to improve the efficiency of all-sky Fourier analyses, it is now also utilized to share sky localization information. When used for this purpose the need for a homogeneous all-sky grid represents a limitation that hinders a broader community adoption. This work presents mhealpy, a Python library able to create, handle and analyze multi-resolution maps, a solution to this problem. It supports efficient pixel querying, arithmetic operation…
The Starlink Hierarchical Data System has been a very successful niche astronomy file format and library for over 30 years. Development of the library was frozen ten years ago when funding for Starlink was stopped and almost no-one remains who understands the implementation details. To ensure the long-term sustainability of the Starlink application software and to make the extensible N-Dimensional Data Format accessible to a broader range of users, we propose to re-implement the HDS library application interface as a layer on top of the Hierarchical Data Format version 5. We present an overvie…
The increasing volumes of astronomical data require practical methods for data exploration, access and visualisation. The Hierarchical Progressive Survey (HiPS) is a HEALPix based scheme that enables a multi-resolution approach to astronomy data from the individual pixels up to the whole sky. We highlight the decisions and approaches that have been taken to make this scheme a practical solution for managing large volumes of heterogeneous data. Early implementors of this system have formed a network of HiPS nodes, with some 250 diverse data sets currently available, with multiple mirror impleme…
We describe a case study to use the Montage image mosaic engine to create maps of the ALLWISE image data set in the Hierarchical Progressive Survey (HiPS) sky-tesselation scheme. Our approach demonstrates that Montage reveals the science content of infrared images in greater detail than has hitherto been possible in HiPS maps. The approach exploits two unique (to our knowledge) characteristics of the Montage image mosaic engine: background modeling to rectify the time variable image backgrounds to common levels; and an adaptive image stretch to present images for visualization. The creation of…
The FITS (Flexible Image Transport System) data format was developed in the late 1970s for storage and exchange of astronomy-related image data. Since then, it has become a standard file format not only for images, but also for radio interferometer data (e.g. UVFITS, FITS-IDI). But is FITS the right format for next-generation telescopes to adopt? The newer Hierarchical Data Format (HDF5) file format offers considerable advantages over FITS, but has yet to gain widespread adoption within radio astronomy. One of the major holdbacks is that HDF5 is not well supported by data reduction software pa…
We introduce the Mechanic, a new open-source code framework. It is designed to reduce the development effort of scientific applications by providing unified API (Application Programming Interface) for configuration, data storage and task management. The communication layer is based on the well-established Message Passing Interface (MPI) standard, which is widely used on variety of parallel computers and CPU-clusters. The data storage is performed within the Hierarchical Data Format (HDF5). The design of the code follows em core-module approach which allows to reduce the user's codebase and mak…
We develop the Mechanic package, which is a new numerical framework for dynamical astronomy. The aim of our software is to help in massive numerical simulations by efficient task management and unified data storage. The code is built on top of the Message Passing Interface (MPI) and Hierarchical Data Format (HDF5) standards and uses the Task Farm approach to manage numerical tasks. It relies on the core-module approach. The numerical problem implemented in the user-supplied module is separated from the host code (core). The core is designed to handle basic setup, data storage and communication…
The FITS (Flexible Image Transport System) data format has been the de facto data format for astronomy-related data products since its inception in the late 1970s. While the FITS file format is widely supported, it lacks many of the features of more modern data serialization, such as the Hierarchical Data Format (HDF5). The HDF5 file format offers considerable advantages over FITS, such as improved I/O speed and compression, but has yet to gain widespread adoption within astronomy. One of the major holdbacks is that HDF5 is not well supported by data reduction software packages and image viewe…
Bayesian parameter estimation is fast becoming the language of gravitational-wave astronomy. It is the method by which gravitational-wave data is used to infer the sources' astrophysical properties. We introduce a user-friendly Bayesian inference library for gravitational-wave astronomy, Bilby. This python code provides expert-level parameter estimation infrastructure with straightforward syntax and tools that facilitate use by beginners. It allows users to perform accurate and reliable gravitational-wave parameter estimation on both real, freely-available data from LIGO/Virgo, and simulated d…
Columbia University in the City of New York（纽约市哥伦比亚大学）；University of Florida（佛罗里达大学）
Gravitational wave astronomy is a vibrant field that leverages both classic and modern data processing techniques for the understanding of the universe. Various approaches have been proposed for improving the efficiency of the detection scheme, with hierarchical matched filtering being an important strategy. Meanwhile, deep learning methods have recently demonstrated both consistency with matched filtering methods and remarkable statistical performance. In this work, we propose Hierarchical Detection Network (HDN), a novel approach to efficient detection that combines ideas from hierarchical m…
We describe a computational framework for hierarchical Bayesian inference with simple (typically single-plate) parametric graphical models that uses graphics processing units (GPUs) to accelerate computations, enabling deployment on very large datasets. Its C++ implementation, CUDAHM (CUDA for Hierarchical Models) exploits conditional independence between instances of a plate, facilitating massively parallel exploration of the replication parameter space using the single instruction, multiple data architecture of GPUs. It provides support for constructing Metropolis-within-Gibbs samplers that …
We present a novel unsupervised learning approach to automatically segment and label images in astronomical surveys. Automation of this procedure will be essential as next-generation surveys enter the petabyte scale: data volumes will exceed the capability of even large crowd-sourced analyses. We demonstrate how a growing neural gas (GNG) can be used to encode the feature space of imaging data. When coupled with a technique called hierarchical clustering, imaging data can be automatically segmented and labelled by organising nodes in the GNG. The key distinction of unsupervised learning is tha…
In the first two years of Gravitational Wave (GW) Astronomy, half a dozen compact binary coalescences (CBCs) have been detected. As the sensitivities and bandwidths of the detectors improve and new detectors join the network, many more sources are expected to be detected. The goal will not only be to find as many sources as possible in the data but to understand the dynamics of the sources much more precisely. Standard searches are currently restricted to a smaller parameter space which assumes aligned spins. Construction of a larger and denser parameter space, and optimising the resultant inc…
Transient radio phenomena and pulsars are one of six LOFAR Key Science Projects (KSPs). As part of the Transients KSP, the Pulsar Working Group (PWG) has been developing the LOFAR Pulsar Data Pipelines to both study known pulsars as well as search for new ones. The pipelines are being developed for the Blue Gene/P (BG/P) supercomputer and a large Linux cluster in order to utilize enormous amounts of computational capabilities (50Tflops) to process data streams of up to 23TB/hour. The LOFAR pipeline output will be using the Hierarchical Data Format 5 (HDF5) to efficiently store large amounts of…
