We report the outcomes of a survey that explores the current practices, needs and expectations of the astrophysics community, concerning four research aspects: open science practices, data access and management, data visualization, and data analysis. The survey, involving 329 professionals from several research institutions, pinpoints significant gaps in matters such as results reproducibility, availability of visual analytics tools and adoption of Machine Learning techniques for data analysis. This research is conducted in the context of the H2020 NEANIAS project.
Interactive visualization and simulation of astrophysical phenomena help astronomers and enable digital planetariums and television documentaries to take their spectators on a journey into deep space to explore the astronomical wonders of our universe in 3D.
The American Astronomical Society's WorldWide Telescope (WWT) project enables terabytes of astronomical images, data, and stories to be viewed and shared among researchers, exhibited in science museums, projected into full-dome immersive planetariums and virtual reality headsets, and taught in classrooms from middle school to college levels. We review the WWT ecosystem, how WWT has been used in the astronomical community, and comment on future directions.
Interactive visualization of astronomical catalogs requires novel techniques due to the huge volumes and complex structure of the data produced by existing and upcoming astronomical surveys. The creation as well as the disclosure of the catalogs can be handled by data pulling mechanisms. These prevent unnecessary processing and facilitate data sharing by having users request the desired end products. In this work we present query driven visualization as a logical continuation of data pulling. Scientists can request catalogs in a declarative way and set process parameters directly from within t…
The Montage Image Mosaic Engine was designed as a scalable toolkit, written in C for performance and portability across *nix platforms, that assembles FITS images into mosaics. The code is freely available and has been widely used in the astronomy and IT communities for research, product generation and for developing next-generation cyber-infrastructure. Recently, it has begun to finding applicability in the field of visualization. This has come about because the toolkit design allows easy integration into scalable systems that process data for subsequent visualization in a browser or client. …
We present AMADA, an interactive web application to analyse multidimensional datasets. The user uploads a simple ASCII file and AMADA performs a number of exploratory analysis together with contemporary visualizations diagnostics. The package performs a hierarchical clustering in the parameter space, and the user can choose among linear, monotonic or non-linear correlation analysis. AMADA provides a number of clustering visualization diagnostics such as heatmaps, dendrograms, chord diagrams, and graphs. In addition, AMADA has the option to run a standard or robust principal components analysis…
Universidad Tecnológica de Chile – INACAP（智利理工大学）；Pontificia Universidad Católica de Chile（智利天主教大学）；Harvard University（哈佛大学）；Millennium Institute of Astrophysics（千禧年天体物理研究所）
Within the last years, the classification of variable stars with Machine Learning has become a mainstream area of research. Recently, visualization of time series is attracting more attention in data science as a tool to visually help scientists to recognize significant patterns in complex dynamics. Within the Machine Learning literature, dictionary-based methods have been widely used to encode relevant parts of image data. These methods intrinsically assign a degree of importance to patches in pictures, according to their contribution in the image reconstruction. Inspired by dictionary-based …
Upcoming and future astronomy research facilities will systematically generate terabyte-sized data sets moving astronomy into the Petascale data era. While such facilities will provide astronomers with unprecedented levels of accuracy and coverage, the increases in dataset size and dimensionality will pose serious computational challenges for many current astronomy data analysis and visualization tools. With such data sizes, even simple data analysis tasks (e.g. calculating a histogram or computing data minimum/maximum) may not be achievable without access to a supercomputing facility. To effe…
The visual inspection of image and catalog data continues to be a valuable aspect of astronomical data analysis. As the scale of astronomical image and catalog data continues to grow, visualizing the data becomes increasingly difficult. In this work, we introduce FitsMap, a simple, lightweight tool for visualizing astronomical image and catalog data. FitsMap only requires a simple web server and can scale to over gigapixel images with tens of millions of sources. Further, the web-based visualizations can be viewed performantly on mobile devices. FitsMap is implemented in Python and is open sou…
Traditional analysis techniques may not be sufficient for astronomers to make the best use of the data sets that current and future instruments, such as the Square Kilometre Array and its Pathfinders, will produce. By utilizing the incredible pattern-recognition ability of the human mind, scientific visualization provides an excellent opportunity for astronomers to gain valuable new insight and understanding of their data, particularly when used interactively in 3D. The goal of our work is to establish the feasibility of a real-time 3D monitoring system for data going into the Australian SKA P…
Wide-angle surveys have been an engine for new discoveries throughout the modern history of astronomy, and have been among the most highly cited and scientifically productive observing facilities in recent years. This trend is likely to continue over the next decade, as many of the most important questions in astrophysics are best tackled with massive surveys, often in synergy with each other and in tandem with the more traditional observatories. We argue that these surveys are most productive and have the greatest impact when the data from the surveys are made public in a timely manner. The r…
We present a high-performance, graphics processing unit (GPU)-based framework for the efficient analysis and visualization of (nearly) terabyte (TB)-sized 3-dimensional images. Using a cluster of 96 GPUs, we demonstrate for a 0.5 TB image: (1) volume rendering using an arbitrary transfer function at 7--10 frames per second; (2) computation of basic global image statistics such as the mean intensity and standard deviation in 1.7 s; (3) evaluation of the image histogram in 4 s; and (4) evaluation of the global image median intensity in just 45 s. Our measured results correspond to a raw computat…
The WorldWide Telescope(WWT) is a scientific visualization platform which can browse deep space images, star catalogs, and planetary remote sensing data from different observation facilities in a three-dimensional virtual scene. First launched and then open-sourced by Microsoft Research, the WWT is now managed by the American Astronomical Society (AAS). Hierarchical Progressive Survey (HiPS) is an astronomical data release scheme proposed by Centre de Données astronomiques de Strasbourg (CDS) and has been accepted as a recommendation by International Virtual Observatory Alliance (IVOA). The Hi…
Data immersion has advantages in astrophysical visualization. Complex multi-dimensional data and phase spaces can be explored in a seamless and interactive viewing environment. Putting the user in the data is a first step toward immersive data analysis. We present a technique for creating 360 degree spherical panoramas with astrophysical data. The three-dimensional software package Blender and the Google Spatial Media module are used together to immerse users in data exploration. Several examples employing these methods exhibit how the technique works using different types of astronomical data…
Punzo et al. (2015) recently reported on the state of the art for visualisation of H I data cubes. I here briefly describe another program, FRELLED, specifically designed for dealing with H I data. Unlike many 3D viewers, FRELLED can handle astronomical world coordinates, easily and interactively mask and label specific volumes within the data, overlay optical data from the SDSS, generate contour plots and renzograms, make basic spectral profile measurements via an interface with MIRIAD, and can switch between viewing the data in 3D and 2D. The code is open source and can potentially be extend…
We present a citation pattern analysis between astronomical papers and 13 other disciplines, based on the arXiv database over the past decade ($2010 - 2020$). We analyze 12,600 astronomical papers citing over 14,531 unique publications outside astronomy. Two striking patterns are unraveled. First, general relativity recently became the most cited field by astronomers, a trend highly correlated with the discovery of gravitational waves. Secondly, the fast growth of referenced papers in computer science and statistics, the first with a notable 15-fold increase since 2015. Such findings confirm t…
The ever-growing datasets in observational astronomy have challenged scientists in many aspects, including an efficient and interactive data exploration and visualization. Many tools have been developed to confront this challenge. However, they usually focus on displaying the actual images or focus on visualizing patterns within catalogs in a predefined way. In this paper we introduce Vizic, a Python visualization library that builds the connection between images and catalogs through an interactive map of the sky region. Vizic visualizes catalog data over a custom background canvas using the s…
Virtual Observatories (VOs) are online hubs of scientific knowledge. They encompass a collection of platforms dedicated to the storage and dissemination of astronomical data, from simple data archives to e-research platforms offering advanced tools for data exploration and analysis. Whilst the more mature platforms within VOs primarily serve the observational community, there are also services fulfilling a similar role for theoretical data. Scientific visualization can be an effective tool for analysis and exploration of datasets made accessible through web platforms for theoretical data, whic…
As the size of images and data products derived from astronomical data continues to increase, new tools are needed to visualize and interact with that data in a meaningful way. Motivated by our own astronomical images taken with the Dark Energy Camera (DECam) we present Toyz, an open source Python package for viewing and analyzing images and data stored on a remote server or cluster. Users connect to the Toyz web application via a web browser, making it an convenient tool for students to visualize and interact with astronomical data without having to install any software on their local machine…
The next decade will feature a growing number of massive ground-based photometric, spectroscopic, and time-domain surveys, including those produced by DECam, DESI, and LSST. The NOAO Data Lab was launched in 2017 to enable efficient exploration and analysis of large surveys, with particular focus on the petabyte-scale holdings of the NOAO Archive and their associated catalogs. The Data Lab mission and future development align well with two of the NSF's Big Ideas, namely Harnessing Data for 21st Century Science and Engineering and as part of a network to contribute to Windows on the Universe: T…
3D visualization is an important data analysis and knowledge discovery tool, however, interactive visualization of large 3D astronomical datasets poses a challenge for many existing data visualization packages. We present a solution to interactively visualize larger-than-memory 3D astronomical data cubes by utilizing a heterogeneous cluster of CPUs and GPUs. The system partitions the data volume into smaller sub-volumes that are distributed over the rendering workstations. A GPU-based ray casting volume rendering is performed to generate images for each sub-volume, which are composited to gene…
Research activities during the last decade have shown the strong potential of photonic devices to greatly simplify ground based and space borne astronomical instruments and to improve their performance. We focus specifically on the mid-infrared wavelength regime (about 5-20 microns), a spectral range offering access to warm objects (about 300 K) and to spectral features that can be interpreted as signatures for biological activity (e.g. water, ozone, carbon dioxide). We review the relevant research activities aiming at the development of single-mode guided optics and the corresponding manufact…
The time domain has been identified as one of the most important areas of astronomical research for the next decade. The Virtual Observatory is in the vanguard with dedicated tools and services that enable and facilitate the discovery, dissemination and analysis of time domain data. These range in scope from rapid notifications of time-critical astronomical transients to annotating long-term variables with the latest modeling results. In this paper, we will review the prior art in these areas and focus on the capabilities that the VAO is bringing to bear in support of time domain science. In p…
Innovative developments in data processing, archiving, analysis, and visualization are nowadays unavoidable to deal with the data deluge expected in next-generation facilities for radio astronomy, such as the Square Kilometre Array (SKA) and its precursors. In this context, the integration of source extraction and analysis algorithms into data visualization tools could significantly improve and speed up the cataloguing process of large area surveys, boosting astronomer productivity and shortening publication time. To this aim, we are developing a visual analytic platform (CIRASA) for advanced …
We present the design of a novel way of handling astronomical catalogs in Astro-WISE in order to achieve the scalability required for the data produced by large scale surveys. A high level of automation and abstraction is achieved in order to facilitate interoperation with visualization software for interactive exploration. At the same time flexibility in processing is enhanced and data is shared implicitly between scientists. This is accomplished by using a data model that primarily stores how catalogs are derived; the contents of the catalogs are only created when necessary and stored only w…
Astronomy has a rich tradition of using color photography and imaging, for visualization in research as well as for sharing scientific discoveries in formal and informal education settings (i.e., for "public outreach.") In the modern era, astronomical research has benefitted tremendously from electronic cameras that allow data and images to be generated and analyzed in a purely digital form with a level of precision not previously possible. Advances in image-processing software have also enabled color-composite images to be made in ways much more complex than with darkroom techniques, not only…
The rapid growth in scale and complexity of both computational and observational astrophysics over the past decade necessitates efficient and intuitive methods for examining and visualizing large datasets. Here we discuss some newly developed tools to import and manipulate astrophysical data into the three dimensional visual effects software, {\it Houdini}. This software is widely used by visual effects artists, but a recently implemented Python API now allows astronomers to more easily use Houdini as a visualization tool. This paper includes a description of features, work flow, and various e…
The size of astronomical observational data is increasing yearly. For example, while Atacama Large Millimeter/submillimeter Array is expected to generate 200 TB raw data every year, Large Synoptic Survey Telescope is estimated to produce 15 TB raw data every night. Since the increasing rate of computing is much lower than that of astronomical data, to provide high performance computing (HPC) resources together with scientific data will be common in the next decade. However, the installation and maintenance costs of a HPC system can be burdensome for the provider. I note public cloud computing …
We demonstrate that extremely rapid and weak periodic and non-periodic signals can easily be detected by using the autocorrelation of intensity as a function of time. We use standard radio-astronomical observations that have artificial periodic and non-periodic signals generated by the electronics of terrestrial origin. The autocorrelation detects weak signals that have small amplitudes because it averages over long integration times. Another advantage is that it allows a direct visualization of the shape of the signals, while it is difficult to see the shape with a Fourier transform. Although…
Astronomy and astrophysics are witnessing dramatic increases in data volume as detectors, telescopes and computers become ever more powerful. During the last decade, sky surveys across the electromagnetic spectrum have collected hundreds of terabytes of astronomical data for hundreds of millions of sources. Over the next decade, the data volume will enter the petabyte domain, and provide accurate measurements for billions of sources. Astronomy and physics students are not traditionally trained to handle such voluminous and complex data sets. In this paper we describe astroML; an initiative, ba…
Astronomy is entering a new era of discovery, coincident with the establishment of new facilities for observation and simulation that will routinely generate petabytes of data. While an increasing reliance on automated data analysis is anticipated, a critical role will remain for visualization-based knowledge discovery. We have investigated scientific visualization applications in astronomy through an examination of the literature published during the last two decades. We identify the two most active fields for progress - visualization of large-N particle data and spectral data cubes - discuss…
Dome A, Antarctica has been thought to be one of the best astronomical sites on the earth since decades ago. From it was first visited by astronomers in 2008, dozens of facilities for astronomical observation and site testing were deployed. Due to its special geographical location, the data and message exchange between Dome A and the domestic control center could only depend on Iridium. Because the link bandwidth of Iridium is extremely limited, meanwhile the network traffic cost is quite expensive and the network is rather unstable, the commonly used data transfer tools, such as rsync and scp…
Astronomical data take on a multitude of forms -- catalogs, data cubes, images, and simulations. The availability of software for rendering high-quality three-dimensional graphics lends itself to the paradigm of exploring the incredible parameter space afforded by the astronomical sciences. The software program Blender gives astronomers a useful tool for displaying data in a manner used by three-dimensional (3D) graphics specialists and animators. The interface to this popular software package is introduced with attention to features of interest in astronomy. An overview of the steps for gener…
We present a framework to interactively volume-render three-dimensional data cubes using distributed ray-casting and volume bricking over a cluster of workstations powered by one or more graphics processing units (GPUs) and a multi-core CPU. The main design target for this framework is to provide an in-core visualization solution able to provide three-dimensional interactive views of terabyte-sized data cubes. We tested the presented framework using a computing cluster comprising 64 nodes with a total of 128 GPUs. The framework proved to be scalable to render a 204 GB data cube with an average…
Since the early 1990s with the arrival of a variety of new technologies, the capacity for authentic astronomical research at the high school level has skyrocketed. This potential, however, has not realized the bright-eyed hopes and dreams of the early pioneers who expected to revolutionise science education through the use of telescopes and other astronomical instrumentation in the classroom. In this paper, a general history and analysis of these attempts is presented. We define what we classify as an Astronomy Research in the Classroom (ARiC) project and note the major dimensions on which the…
Astronomical data generally consists of 2 or more high-resolution axes, e.g., X,Y position on the sky or wavelength and position-along-one-axis (long-slit spectrometer). Analyzing these multi-dimension observations requires combining 3D source models (including velocity effects), instrument models, and multi-dimensional data comparison and fitting. A prototype of such a "Beyond-XSPEC" (Noble & Nowak, 2008) system is presented here using Chandra imag- ing and dispersed HETG grating data. Techniques used include: Monte Carlo event generation, chi-squared comparison, conjugate gradient fitting ad…
An automated, rapid classification of transient events detected in the modern synoptic sky surveys is essential for their scientific utility and effective follow-up using scarce resources. This presents some unusual challenges: the data are sparse, heterogeneous and incomplete; evolving in time; and most of the relevant information comes not from the data stream itself, but from a variety of archival data and contextual information (spatial, temporal, and multi-wavelength). We are exploring a variety of novel techniques, mostly Bayesian, to respond to these challenges, using the ongoing CRTS s…
Upcoming HI surveys will deliver large datasets, and automated processing using the full 3-D information (two positional dimensions and one spectral dimension) to find and characterize HI objects is imperative. In this context, visualization is an essential tool for enabling qualitative and quantitative human control on an automated source finding and analysis pipeline. We discuss how Visual Analytics, the combination of automated data processing and human reasoning, creativity and intuition, supported by interactive visualization, enables flexible and fast interaction with the 3-D data, helpi…
We report on an exploratory project aimed at performing immersive 3D visualization of astronomical data, starting with spectral-line radio data cubes from galaxies. This work is done as a collaboration between the Department of Physics and Astronomy and the Department of Computer Science at the University of Manitoba. We are building our prototype using the 3D engine Unity, because of its ease of use for integration with advanced displays such as a CAVE environment, a zSpace tabletop, or virtual reality headsets. We address general issues regarding 3D visualization, such as: load and convert a…
We present a set of open-source web tools for visualization of spectral and imaging data, which we use in the second Reference Catalogue of Spectral Energy Distributions of galaxies RCSEDv2 (https://rcsed2.voxastro.org/). Using modern web frameworks Quasar and Vue.js we developed interactive viewers to visualize spectra and SEDs of galaxies and the diagrams presenting emission line ratios determined from the analysis of their spectra (BPT diagrams). The viewers are built in Javascript which puts a minimum load on the server side while providing full interactivity for the user. The use of moder…
The FITS file format has become the de facto standard for sharing, analyzing, and archiving astronomy data over the last four decades. FITS was adopted by astronomers in the early 1980s to overcome incompatibilities between operating systems. On the back of FITS' success, astronomical data became both backwards compatible and easily shareable. However, new advances in astronomical instrumentation, computational technologies, and analytic techniques have resulted in new data that do not work well within the traditional FITS format. Tensions have arisen between the desire to update the format to…
SKA precursors are capable of detecting hundreds of galaxies in HI in a single 12 hours pointing. In deeper surveys one will probe more easily faint HI structures, typically located in the vicinity of galaxies, such as tails, filaments, and extraplanar gas. The importance of interactive visualization has proven to be fundamental for the exploration of such data as it helps users to receive immediate feedback when manipulating the data. We have developed SlicerAstro, a 3-D interactive viewer with new analysis capabilities, based on traditional 2-D input/output hardware. These capabilities enhan…
Micro-electro-mechanical systems (MEMS) technology can provide for deformable mirrors (DMs) with excellent performance within a favorable economy of scale. Large MEMS-based astronomical adaptive optics (AO) systems such as the Gemini Planet Imager are coming on-line soon. As MEMS DM end-users, we discuss our decade of practice with the micromirrors, from inspecting and characterizing devices to evaluating their performance in the lab. We also show MEMS wavefront correction on-sky with the "Villages" AO system on a 1-m telescope, including open-loop control and visible-light imaging. Our work d…
Over the past decades and even centuries, the astronomical community has accumulated a signif-icant heritage of recorded observations of a great many astronomical objects. Those records con-tain irreplaceable information about long-term evolutionary and non-evolutionary changes in our Universe, and their preservation and digitization is vital. Unfortunately, most of those data risk becoming degraded and thence totally lost. We hereby call upon the astronomical community and US funding agencies to recognize the gravity of the situation, and to commit to an interna-tional preservation and digiti…
The American Astronomical Society (AAS) Journals are a vital asset of our professional society. With the push towards open access, page charges are a viable and sustainable option for continuing to effectively fund and publish the AAS Journals. However, the existing page charge model, which requires individual authors to pay page charges out of their grants or even out of pocket, is already challenging to some researchers and could be exacerbated in the Open Access (OA) era if charges increase. A discussion of alternative models for funding page charges and publishing costs should be part of t…
Astronomical images suffer a constant presence of multiple defects that are consequences of the intrinsic properties of the acquisition equipments, and atmospheric conditions. One of the most frequent defects in astronomical imaging is the presence of additive noise which makes a denoising step mandatory before processing data. During the last decade, a particular modeling scheme, based on sparse representations, has drawn the attention of an ever growing community of researchers. Sparse representations offer a promising framework to many image and signal processing tasks, especially denoising…
Splotch is a rendering algorithm for exploration and visual discovery in particle-based datasets coming from astronomical observations or numerical simulations. The strengths of the approach are production of high quality imagery and support for very large-scale datasets through an effective mix of the OpenMP and MPI parallel programming paradigms. This article reports our experiences in re-designing Splotch for exploiting emerging HPC architectures nowadays increasingly populated with GPUs. A performance model is introduced for data transfers, computations and memory access, to guide our re-f…
Astronomy has entered the multi-messenger data era and Machine Learning has found widespread use in a large variety of applications. The exploitation of synoptic (multi-band and multi-epoch) surveys, like LSST (Legacy Survey of Space and Time), requires an extensive use of automatic methods for data processing and interpretation. With data volumes in the petabyte domain, the discrimination of time-critical information has already exceeded the capabilities of human operators and crowds of scientists have extreme difficulty to manage such amounts of data in multi-dimensional domains. This work i…
In this decade astronomy is undergoing a paradigm shift to handle data from next generation observatories such as the Square Kilometre Array (SKA) or the Vera C. Rubin Observatory (LSST). Producing real time data streams of up to 10 TB/s and data products of the order of 600 Pbytes/year, the SKA will be the biggest civil data producing machine of the world that demands novel solutions on how these data volumes can be stored and analysed. Through the use of complex, automated pipelines the provenance of this real time data processing is key to establish confidence within the system, its final d…
Data volumes from multiple sky surveys have grown from gigabytes into terabytes during the past decade, and will grow from terabytes into tens (or hundreds) of petabytes in the next decade. This exponential growth of new data both enables and challenges effective astronomical research, requiring new approaches. Thus far, astronomy has tended to address these challenges in an informal and ad hoc manner, with the necessary special expertise being assigned to e-Science or survey science. However, we see an even wider scope and therefore promote a broader vision of this data-driven revolution in a…
