Data challenges as a tool for time-domain astronomy
Renée Hložek
备注：
Accepted for publication in the Publications of the Astronomical Society of the Pacific as part of a 'Focus on Tools and Techniques for Time-domain Astronomy'
Data challenges are emerging as powerful tools with which to answer fundamental astronomical questions. Time-domain astronomy lends itself to data challenges, particularly in the era of classification and anomaly detection. With improved sensitivity of wide-field surveys in optical and radio wavelengths from surveys like the Large Synoptic Survey Telescope (LSST) and the Canadian Hydrogen Intensity Mapping Experiment (CHIME), we are entering the large-volume era of transient astronomy. I highlight some recent time-domain challenges, with particular focus on the Photometric LSST Astronomical Time series Classification Challenge (PLAsTiCC), and describe metrics used to evaluate the performance of those entering data challenges.
We present an Open Educational Resource system that instructors can use to assign automatically graded, randomized, and scaffolded assessment questions for astronomy classes.
In the context of astronomy projects, scientists have been confronted with the problem of describing in a standardized way how their data have been produced. As presented in a talk at last year's ADASS, the International Virtual Observatory Alliance (IVOA) is working on the definition of a Provenance Data Model, compatible with the W3C PROV model, which shall describe how provenance metadata can be modeled, stored and exchanged in astronomy. In this poster, we present the current status of our developments of libraries and tools, mainly open source, which implement the IVOA Provenance Data Mod…
We propose a new method for solving an important problem of astronomy that arises in observations with ultrahigh-angular-resolution interferometers. This method is based on the application of the theory of artificial neural networks. We propose and compute a multiparameter model for a celestial object like Sgr A*. For this model we have numerically constructed a number of probable images for neural network training. After neural network training on these images, the quality of its operation has been tested on another series of images from the same model. We have proven that a neural network ca…
Common analysis techniques in multi-messenger astronomy involve hypothesis tests with unbinned log-likelihood (LLH) functions using data recorded in celestial coordinates to identify sources of high-energy cosmic particles in the Universe. We present the new Python-based tool "SkyLLH" to develop such analyses in a telescope-independent framework. The main goal of the software is to provide an easy-to-use and modularized concept to implement and to execute such LLH functions efficiently on the computer with high-performance. SkyLLH can be applied on different multi-messenger data like neutrino …
The large infrastructure projects for the next decade will allow a new quantum leap in terms of new possible science. ESFRI, the European Strategy Forum on Research Infrastructures, a strategic initiative to develop the scientific integration of Europe, has identified four facilities (SKA, CTA, KM3Net and E-ELT) deserving priority in support. The ASTERICS project aims to address the cross-cutting synergies and common challenges shared by the various Astronomy ESFRI and other world-class facilities. The project (22 partners across Europe) is funded by the EU Horizon 2020 programme with 15 MEuro…
Since its inception in the early 2000, the Virtual Observatory (VO), developed as a collaboration of many national and international projects, has become a major factor in the discovery and dissemination of astronomical information worldwide. The IVOA has been coordinating all these efforts worldwide to ensure a common VO framework that enables transparent access to and interoperability of astronomy resources (data and software) around the world. The VO is not a magic solution to all astronomy data management challenges but it does bring useful solutions in many areas borne out by the fact tha…
MASER (Measurements, Analysis, and Simulation of Emission in the Radio range) is a comprehensive infrastructure dedicated to time-dependent low frequency radio astronomy (up to about 50 MHz). The main radio sources observed in this spectral range are the Sun, the magnetized planets (Earth, Jupiter, Saturn), and our Galaxy, which are observed either from ground or space. Ground observatories can capture high resolution data streams with a high sensitivity. Conversely, space-borne instruments can observe below the ionospheric cut-off (at about 10 MHz) and can be placed closer to the studied obje…
We occasionally include projects in our learner-centered introductory astronomy college course to enable non-science major students explore some astronomical concepts in more detail than otherwise. Such projects also highlight ongoing or upcoming astronomical events. We hope that students will feel more interested in astronomy through projects tied to astronomical events. In Spring 2012, we offered short essays focused on eclipses, transits and occultations to promote the rare transit of Venus that occurred on June 5th, 2012. We asked students to write two short essays from three that were off…
The increase in the number of researchers coupled with the ease of publishing and distribution of scientific papers (due to technological advancements) has resulted in a dramatic increase in astronomy literature. This has likely led to the predicament that the body of the literature is too large for traditional human consumption and that related and crucial knowledge is not discovered by researchers. In addition to the increased production of astronomical literature, recent decades have also brought several advancements in computational linguistics. Especially, the machine-aided processing of …
The use of Python is noticeably growing among the scientific community, and Astronomy is not an exception. The power of Python consists of being an extremely versatile high-level language, easy to program that combines both traditional programming and data reduction and analysis tools. Here I make a brief introduction to Python, mentioning a few programming practices implemented in the language and some of its useful features on the process of data manipulation. I cover in a little more detail the standard scientific libraries (NumPy and SciPy) for data handling, the graphical library (Matplot…
The problem of denoising a one-dimensional signal possessing varying degrees of smoothness is ubiquitous in time-domain astronomy and astronomical spectroscopy. For example, in the time domain, an astronomical object may exhibit a smoothly varying intensity that is occasionally interrupted by abrupt dips or spikes. Likewise, in the spectroscopic setting, a noiseless spectrum typically contains intervals of relative smoothness mixed with localized higher frequency components such as emission peaks and absorption lines. In this work, we present trend filtering, a modern nonparametric statistical…
We present a user-friendly, but powerful interface for the data mining of scientific repositories. We present the tool in use with actual astronomy data and show how it may be used to achieve many different types of powerful semantic queries. The tool itself hides the gory details of query formulation, and data retrieval from the user, and allows the user to create workflows which may be used to transform the data into a convenient form.
In the last decade a new generation of telescopes and sensors has allowed the production of a very large amount of data and astronomy has become, a data-rich science; this transition is often labeled as: "data revolution" and "data tsunami". The first locution puts emphasis on the expectations of the astronomers while the second stresses, instead, the dramatic problem arising from this large amount of data: which is no longer computable with traditional approaches to data storage, data reduction and data analysis. In a new, age new instruments are necessary, as it happened in the Bronze age wh…
Astronomy has been at the forefront of the development of the techniques and methodologies of data intensive science for over a decade with large sky surveys and distributed efforts such as the Virtual Observatory. However, it faces a new data deluge with the next generation of synoptic sky surveys which are opening up the time domain for discovery and exploration. This brings both new scientific opportunities and fresh challenges, in terms of data rates from robotic telescopes and exponential complexity in linked data, but also for data mining algorithms used in classification and decision ma…
PESIT South Campus（PESIT南校区）；Jain University（贾因大学）
Astroinformatics is a new impact area in the world of astronomy, occasionally called the final frontier, where several astrophysicists, statisticians and computer scientists work together to tackle various data intensive astronomical problems. Exponential growth in the data volume and increased complexity of the data augments difficult questions to the existing challenges. Classical problems in Astronomy are compounded by accumulation of astronomical volume of complex data, rendering the task of classification and interpretation incredibly laborious. The presence of noise in the data makes ana…
The field of astronomy is starting to generate more data than can be managed, served and processed by current techniques. This paper has outlined practices for developing next-generation tools and techniques for surviving this data tsunami, including rigorous evaluation of new technologies, partnerships between astronomers and computer scientists, and training of scientists in high-end software engineering engineering skills.
Increasingly there is a need to develop astronomical visualisation and manipulations tools which allow viewers to interact with displayed data directly, in real time and across a range of platforms. In addition, increases in dynamic range available for astronomical images with next generation telescopes have led to a desire to develop enhanced visualisations capable of presenting information across a wide range of intensities. This paper describes three new tools for astronomical visualisation and image manipulation that are the result of a collaboration between software engineers and radio as…
Several different controlled vocabularies have been developed and used by the astronomical community, each designed to serve a specific need and a specific group. The Unified Astronomy Thesaurus (UAT) attempts to provide a highly structured controlled vocabulary that will be relevant and useful across the entire discipline, regardless of content or platform. As two major use cases for the UAT include classifying articles and data, we examine the UAT in comparison with the Astronomical Subject Keywords used by major publications and the JWST Science Keywords used by STScI's Astronomer's Proposa…
Recently the International Virtual Observatory Alliance (IVOA) released a standard to structure provenance metadata, and several implementations are in development in order to capture, store, access and visualize the provenance of astronomy data products. This BoF will be focused on practical needs for provenance in astronomy. A growing number of projects express the requirement to propose FAIR data (Findable, Accessible, Interoperable and Reusable) and thus manage provenance information to ensure the quality, reliability and trustworthiness of this data. The concepts are in place, but now, ap…
We present a review of data types and statistical methods often encountered in astronomy. The aim is to provide an introduction to statistical applications in astronomy for statisticians and computer scientists. We highlight the complex, often hierarchical, nature of many astronomy inference problems and advocate for cross-disciplinary collaborations to address these challenges.
The operation of the future Cherenkov Telescope Array (CTA), the next generation of Imaging Atmospheric Cherenkov Telescopes (IACTs), as an Open Observatory requires data products and analysis tools to be accessible and usable by the scientific community. This requirement has impelled gamma-ray astronomers to develop open-source science tools and standardised data formats. The objective of this presentation is to provide a perspective on the current effort to define a prototypical format for high-level gamma-ray astronomical data. The initiative, started by the IACT community in 2016, has gain…
We review the current state of data mining and machine learning in astronomy. 'Data Mining' can have a somewhat mixed connotation from the point of view of a researcher in this field. If used correctly, it can be a powerful approach, holding the potential to fully exploit the exponentially increasing amount of available data, promising great scientific advance. However, if misused, it can be little more than the black-box application of complex computing algorithms that may give little physical insight, and provide questionable results. Here, we give an overview of the entire data mining proce…
The Gravitational Wave (GW) universe contains a wealth of sources which, with the proper treatment, will open up the universe as never before. By observing massive black hole binaries to high redshifts, we should begin to explore the formation process of seed black holes and track galactic evolution to the present day. Observations of extreme mass ratio inspirals will allow us to explore galactic centers in the local universe, as well as providing tests of General Relativity and constraining the value of Hubble's constant. The detection of compact binaries in our own galaxy may allow us to mod…
In this work, we identify elements of effective machine learning datasets in astronomy and present suggestions for their design and creation. Machine learning has become an increasingly important tool for analyzing and understanding the large-scale flood of data in astronomy. To take advantage of these tools, datasets are required for training and testing. However, building machine learning datasets for astronomy can be challenging. Astronomical data is collected from instruments built to explore science questions in a traditional fashion rather than to conduct machine learning. Thus, it is of…
Gravitational waves are radiative solutions of space-time dynamics predicted by Einstein's theory of General Relativity. A world-wide array of large-scale and highly sensitive interferometric detectors constantly scrutinizes the geometry of the local space-time with the hope to detect deviations that would signal an impinging gravitational wave from a remote astrophysical source. Finding the rare and weak signature of gravitational waves buried in non-stationary and non-Gaussian instrument noise is a particularly challenging problem. We will give an overview of the data-analysis techniques and…
Astronomy is experiencing a rapid growth in data size and complexity. This change fosters the development of data-driven science as a useful companion to the common model-driven data analysis paradigm, where astronomers develop automatic tools to mine datasets and extract novel information from them. In recent years, machine learning algorithms have become increasingly popular among astronomers, and are now used for a wide variety of tasks. In light of these developments, and the promise and challenges associated with them, the IAC Winter School 2018 focused on big data in Astronomy, with a pa…
The gravitational wave window onto the universe will open in roughly five years, when Advanced LIGO and Virgo achieve the first detections of high frequency gravitational waves, most likely coming from compact binary mergers. Electromagnetic follow-up of these triggers, using radio, optical, and high energy telescopes, promises exciting opportunities in multi-messenger time domain astronomy. In the next decade, space-based observations of low frequency gravitational waves from massive black hole mergers, and their electromagnetic counterparts, will open up further vistas for discovery. This tw…
There is a growing trend toward using high-level tools for design and implementation of radio astronomy digital signal processing (DSP) systems. Such tools, for example, those from the Collaboration for Astronomy Signal Processing and Electronics Research (CASPER), are usually platform-specific, and lack high-level, platform-independent, portable, scalable application specifications. This limits the designer's ability to experiment with designs at a high-level of abstraction and early in the development cycle. We address some of these issues using a model-based design approach employing datafl…
We developed a GPU based single-pulse search pipeline (GSP) with candidate-archiving database. Largely based upon the infrastructure of Open source pulsar search and analysis toolkit (PRESTO), GSP implements GPU acceleration of the de-dispersion and integrates a candidate-archiving database. We applied GSP to the data streams from the commensal radio astronomy FAST survey (CRAFTS), which resulted in a quasi-real-time processing. The integrated candidate database facilitates synergistic usage of multiple machine-learning tools and thus improves efficient identification of radio pulsars such as …
The International Astronomical Youth Camp has benefited thousands of lives during its 50-year history. We explore the pedagogy behind this success, review a survey taken by more than 300 previous participants, and discuss some of the challenges the camp faces in the future.
Astronomy is entering a new era of discovery, coincident with the establishment of new facilities for observation and simulation that will routinely generate petabytes of data. While an increasing reliance on automated data analysis is anticipated, a critical role will remain for visualization-based knowledge discovery. We have investigated scientific visualization applications in astronomy through an examination of the literature published during the last two decades. We identify the two most active fields for progress - visualization of large-N particle data and spectral data cubes - discuss…
We apply a combination of a Genetic Algorithms (GA) and Support Vector Machines (SVM) machine learning algorithm to solve two important problems faced by the astronomical community: star/galaxy separation, and photometric redshift estimation of galaxies in survey catalogs. We use the GA to select the relevant features in the first step, followed by optimization of SVM parameters in the second step to obtain an optimal set of parameters to classify or regress, in process of which we avoid over-fitting. We apply our method to star/galaxy separation in Pan-STARRS1 data. We show that our method co…
The global climate crisis poses new risks to humanity, and with them, new challenges to the practices of professional astronomy. Avoiding the more catastrophic consequences of global warming by more than 1.5 degrees requires an immediate reduction of greenhouse gas emissions. According to the 2018 United Nations Intergovernmental Panel report, this will necessitate a 45% reduction of emissions by 2030 and net-zero emissions by 2050. Efforts are required at all levels, from the individual to the governmental, and every discipline must find ways to achieve these goals. This will be especially di…
For centuries, the rich nocturnal environment of the starry sky could be modelled only by analogue tools such as paper planispheres, atlases, globes and numerical tables. The immersive sky simulator of the twentieth century, the optomechanical planetarium, provided new ways for representing and teaching about the sky, but the high construction and running costs meant that they have not become common. However, in recent decades, "desktop planetarium programs" running on personal computers have gained wide attention. Modern incarnations are immensely versatile tools, mostly targeted towards the …
We spoke with four researchers to understand the accessibility challenges in astronomy research, education and outreach for blind and visually impaired (BVI) persons, as well as solutions to these challenges and how it innovates data analysis methods for all astronomers. Those interviewed: Nicolas Bonne (University of Portsmouth); Cheryl Fogle-Hatch (Museum Senses); Garry Foran (Swinburne University of Technology) and Enrique Perez Montero (Instituto de Astrofísica de Andalucía).
Collaborations in astronomy and astrophysics are faced with numerous cyber infrastructure challenges, such as large data sets, the need to combine heterogeneous data sets, and the challenge to effectively collaborate on those large, heterogeneous data sets with significant processing requirements and complex science software tools. The cyberhubs system is an easy-to-deploy package for small to medium-sized collaborations based on the Jupyter and Docker technology, that allows web-browser enabled, remote, interactive analytic access to shared data. It offers an initial step to address these cha…
Harvard-Smithsonian Center for Astrophysics（哈佛-史密松天体物理中心）
The existing search tools for exploring the NASA Astrophysics Data System (ADS) can be quite rich and empowering (e.g., similar and trending operators), but researchers are not yet allowed to fully leverage semantic search. For example, a query for "results from the Planck mission" should be able to distinguish between all the various meanings of Planck (person, mission, constant, institutions and more) without further clarification from the user. At ADS, we are applying modern machine learning and natural language processing techniques to our dataset of recent astronomy publications to train …
In a recent paper, published at arXiv:0910.2381, we started a discussion on the new possibilities arising from the use of fractional differential calculus in image processing. We have seen that the fractional calculation is able to enhance the quality of images, with interesting possibilities in edge detection and image restoration. Here, we want to discuss more deeply its role as a tool for the processing of astronomical images. In particular, the fractional differentiation can help produce a 'content-matter' based image from a pretty astronomical image that can be used for more research and …
We report on an exploratory project aimed at performing immersive 3D visualization of astronomical data, starting with spectral-line radio data cubes from galaxies. This work is done as a collaboration between the Department of Physics and Astronomy and the Department of Computer Science at the University of Manitoba. We are building our prototype using the 3D engine Unity, because of its ease of use for integration with advanced displays such as a CAVE environment, a zSpace tabletop, or virtual reality headsets. We address general issues regarding 3D visualization, such as: load and convert a…
ESASky is a new science-driven discovery portal for all ESA astronomical missions that gives users worldwide a simplified access to high-level science-ready products from ESA and other data providers. The tool features a sky exploration interface and a single/multiple target interface, and it requires no prior knowledge of specific details of each mission. Users can explore the sky in multiple wavelengths, quickly see the data available for their targets, and retrieve the relevant products, with just a few clicks. The first version of the tool, released in May 2016, provides access to imaging …
TOPCAT, the Tool for OPerations on Catalogues And Tables, is an interactive desktop application for retrieval, analysis and manipulation of tabular data, offering a powerful and flexible range of interactive visualization options amongst other features. Its visualization capabilities focus on enabling interactive exploration of large static local tables - millions of rows and hundreds of columns can easily be handled on a standard desktop or laptop machine, and various options are provided for meaningful graphical representation of such large datasets. TOPCAT has been developed in the context …
Astrophysics and cosmology are rich with data. The advent of wide-area digital cameras on large aperture telescopes has led to ever more ambitious surveys of the sky. Data volumes of entire surveys a decade ago can now be acquired in a single night and real-time analysis is often desired. Thus, modern astronomy requires big data know-how, in particular it demands highly efficient machine learning and image analysis algorithms. But scalability is not the only challenge: Astronomy applications touch several current machine learning research questions, such as learning from biased data and dealin…
Dome A in Antarctica has been demonstrated to be the best site on earth for optical, infrared, and terahertz astronomical observations by more and more evidence, such as excellent free-atmosphere seeing, extremely low perceptible water vapor, low sky background, and continuous dark time, etc. In this review paper, we present a complete picture of the development of astronomy at Dome A from the very beginning, review recent progress in time-domain astronomy, demonstrate exciting results of the site testing, and address the challenges in instrumentation. Currently proposed projects are briefly d…
A major challenge in modern radio astronomy is dealing with the massive data volumes generated by wide-bandwidth receivers. Such massive data rates are often too great for a single device to cope, and so processing must be split across multiple devices working in parallel. These devices must work in unison to process incoming data in real time, reduce the data volume to a manageable size, and output a science-ready data product. The aim of this chapter is to give a broad overview of how digital systems for radio telescopes are commonly implemented, with a focus on real-time stream processing o…
IAU Inter-Commission B2-C1-C2 WG Data-driven Astronomy Education and Public Outreach (DAEPO) was launched officially in April 2017. With the development of many mega-science astronomical projects, for example CTA, DESI, EUCLID, FAST, GAIA, JWST, LAMOST, LSST, SDSS, SKA, and large scale simulations, astronomy has become a Big Data science. Astronomical data is not only necessary resource for scientific research, but also very valuable resource for education and public outreach (EPO), especially in the era of Internet and Cloud Computing. IAU WG Data-driven Astronomy Education and Public Outreac…
In the distant past, astronomy was often intertwined with religion into a unified cosmos. As science became a distinct cultural enterprise, astronomy has witnessed a variety of rich interactions with other fields. Mathematical statistics was stimulated in the 19th century by astronomical problems, and today astrostatistics is a small but growing cross-disciplinary field advancing methodology to address challenges in astronomical data analysis. Throughout the 20th century, astronomy became closely allied with physics such that astronomy and astrophysics are now profoundly intertwined. Physical …
National Research Council Herzberg Institute of Astrophysics（加拿大国家研究委员会赫兹伯格天体物理研究所）
Astronomy is increasingly encountering two fundamental truths: (1) The field is faced with the task of extracting useful information from extremely large, complex, and high dimensional datasets; (2) The techniques of astroinformatics and astrostatistics are the only way to make this tractable, and bring the required level of sophistication to the analysis. Thus, an approach which provides these tools in a way that scales to these datasets is not just desirable, it is vital. The expertise required spans not just astronomy, but also computer science, statistics, and informatics. As a computer sc…
Where do cosmic X-rays come from? Every new unidentified X-ray source could potentially revolutionize our understanding of the universe. The international collaborative astronomy project EXTraS aimed at automatically classifying new sources of X-ray emission (e.g., stars or galaxies) in the large observation database of the X-ray satellite XMM-Newton. Because data archives have reached dimensions of big data astronomers used different machine-learning (ML) random forest decision tree algorithms that performed the classification process. In this bachelor thesis in information design, I was inte…
