The importance of archival science increases significantly for astrophysical observatories as they mature and their archive holdings grow in size and complexity. Further enhancing the science return of archival data requires engaging a larger audience than the mission reference community, mostly because of the growth of interest in multi-wavelength and transient/time variability research. Such a goal, though, can be difficult to achieve. In this paper I will describe a different approach to this question that, while minimizing technological friction and leveraging existing services, makes archival observations more accessible and increases our capability to proactively engage astronomers on potentially interesting archival records. Inspired by this strategy, the Chandra Data Archive team is working on two specific experimental projects that will hopefully demonstrate their potential while contributing to the maximization of the scientific return of the Chandra mission.
In this article I present IEAD, a new interface for astronomical science databases. It is based on a powerful, yet simple, syntax designed to completely abstract the user from the structure of the underlying database. The programming language chosen for its implementation, JavaScript, makes it possible to interact directly with the user and to provide real-time information on the parsing process, error messages, and the name resolution of targets; additionally, the same parsing engine is used for context-sensitive autocompletion. Ultimately, this product should significantly simplify the use o…
The Virtual Observatory (VO) is becoming the de-facto standard for astronomical data publication. However, the number of radio astronomical archives is still low in general, and even lower is the number of radio astronomical data available through the VO. In order to facilitate the building of new radio astronomical archives, easing at the same time their interoperability with VO framework, we have developed a VO-compliant data model which provides interoperable data semantics for radio data. That model, which we call the Radio Astronomical DAta Model for Single-dish (RADAMS) has been built us…
Astronomical observation data require long-term preservation, and the rapid accumulation of observation data makes it necessary to consider the cost of long-term archive storage. In addition to low-speed disk-based online storage, optical disk or tape-based offline storage can be used to save costs. However, for astronomical research that requires historical data (particularly time-domain astronomy), the performance and energy consumption of data-accessing techniques cause problems because the requested data (which are organized according to observation time) may be located across multiple sto…
Data archiving is one of the most critical issues for modern astronomical observations. With the development of a new generation of radio telescopes, the transfer and archiving of massive remote data have become urgent problems to be solved. Herein, we present a practical and robust file-level flow-control approach, called the Unlimited Sliding-Window (USW), by referring to the classic flow-control method in TCP protocol. Basing on the USW and the Next Generation Archive System (NGAS) developed for the Murchison Widefield Array telescope, we further implemented an enhanced archive system (ENGA…
The size of astronomical observational data is increasing yearly. For example, while Atacama Large Millimeter/submillimeter Array is expected to generate 200 TB raw data every year, Large Synoptic Survey Telescope is estimated to produce 15 TB raw data every night. Since the increasing rate of computing is much lower than that of astronomical data, to provide high performance computing (HPC) resources together with scientific data will be common in the next decade. However, the installation and maintenance costs of a HPC system can be burdensome for the provider. I note public cloud computing …
The world astronomical image archives represent huge opportunities to time-domain astronomy sciences and other hot topics such as space defense, and astronomical observatories should improve this wealth and make it more accessible in the big data era. In 2010 we introduced the Mega-Archive database and the Mega-Precovery server for data mining images containing Solar system bodies, with focus on near Earth asteroids (NEAs). This paper presents the improvements and introduces some new related data mining tools developed during the last five years. Currently, the Mega-Archive has indexed 15 mill…
Sky coverage is one of the most important pieces of information about astronomical observations. We discuss possible representations, and present algorithms to create and manipulate shapes consisting of generalized spherical polygons with arbitrary complexity and size on the celestial sphere. This shape specification integrates well with our Hierarchical Triangular Mesh indexing toolbox, whose performance and capabilities are enhanced by the advanced features presented here. Our portable implementation of the relevant spherical geometry routines comes with wrapper functions for database querie…
Archives of Photographic PLates for Astronomical USE (APPLAUSE) Digitisation of astronomical plates and their integration into the International Virtual Observatory
Science archives are cornerstones of modern astronomical facilities. In this paper we describe the version 1.0 milestone of the Atacama Large Millimeter/submillimeter Array Science Archive. This version features a comprehensive query interface with rich metadata and visualisation of the spatial and spectral locations of the observations, a complete set of virtual observatory services for programmatic access, text-based similarity search, display and query for types of astronomical objects in SIMBAD and NED, browser-based remote visualisation, interactive previews with tentative line identifica…
From the moment astronomical observations are made the resulting data products begin to grow stale. Even if perfect binary copies are preserved through repeated timely migration to more robust storage media, data standards evolve and new tools are created that require different kinds of data or metadata. The expectations of the astronomical community change even if the data do not. We discuss data engineering to mitigate the ensuing risks with examples from a recent project to refactor seven million archival images to new standards of nomenclature, metadata, format, and compression.
In the Virtual Observatory era, where we intend to expose scientists (or software agents on their behalf) to a stream of observations from all existing facilities, the ability to access and to further interpret the origin, relationships, and processing steps on archived astronomical assets (their Provenance) is a requirement for proper observation selection, and quality assessment. In this article we present the different use cases Data Provenance is needed for, the challenges inherent to building such a system for the ESO archive, and their link with ongoing work in the International Virtual …
The Italian center for Astronomical Archives tries to provide astronomical data resources as interoperable services based on IVOA standards. Its VO expertise and knowledge comes from active participation within IVOA and VO at European and international level, with a double-fold goal: learn from the collaboration and provide inputs to the community. The first solution to build an easy to configure and maintain resource publisher conformant to VO standards proved to be too optimistic. For this reason it has been necessary to re-think the architecture with a modular system built around the messag…
As the cyber-infrastructure for Astronomical research from Chinese Virtual Observatory (China-VO) project, AstroCloud has been archived solid progresses during the last one year. Proposal management system and data access system are re-designed. Several new sub-systems are developed, including China-VO PaperData, AstroCloud Statics and Public channel. More data sets and application environments are integrated into the platform. LAMOST DR1, the largest astronomical spectrum archive was released to the public using the platform. The latest progresses will be introduced.
INAF Trieste Astronomical Observatory (OATs) has a long tradition in information technology applied to Astronomical and Astrophysical use cases, particularly for what regards computing for data reduction, analysis and simulations; data and archives management; space missions data processing; design and software development for ground-based instruments. The ensemble of these activities, in the last years, pushed the need to acquire new computing resources and technologies and to deep competences in theirs management. In this paper we describe INAF-OATs computing centre technological stuff, our …
The Liverpool Telescope has been in fully autonomous operation since 2004. The supporting data archive facility has largely been untouched. The data provision service has not been an issue although some modernisation of the system is desirable. This project is timely. Not only does it suit the upgrade of the current LT data archive, it is in line with the design phase of the New Robotic Telescope which will be online in the early-2020s; and with the development of a new data archive facility for a range of telescopes at the National Astronomical Research Institute of Thailand. The Newton Fund …
ESA Gaia mission is producing the more accurate source catalogue in astronomy up to now. That represents a challenge on the archiving area to make accessible this information to the astronomers in an efficient way. Also, new astronomical missions have reinforced the change on the development of archives. Archives, as simple applications to access the data are being evolving into complex data center structures where computing power services are available for users and data mining tools are integrated into the server side. In the case of astronomy science that involves the use of big catalogues,…
The data volumes stored in telescope archives is constantly increasing due to the development and improvements in the instrumentation. Often the archives need to be stored over a distributed storage architecture, provided by independent compute centres. Such a distributed data archive requires overarching data management orchestration. Such orchestration comprises of tools which handle data storage and cataloguing, and steering transfers integrating different storage systems and protocols, while being aware of data policies and locality. In addition, it needs a common Authorisation and Authent…
In the era of big data astronomy, next generation telescopes and large sky surveys produce data sets at the TB or even PB level. Due to their large data volumes, these astronomical data sets are extremely difficult to transfer and analyze using personal computers or small clusters. In order to offer better access to data, data centers now generally provide online science platforms that enable analysis close to the data. The Chinese Virtual Observatory (China-VO) is one of the member projects in the International Virtual Observatory Alliance and it is dedicated to providing a research and educa…
We present a novel approach to quality control during the processing of astronomical data. Quality control in the Astro-WISE Information System is integral to all aspects of data handing and provides transparent access to quality estimators for all stages of data reduction from the raw image to the final catalog. The implementation of quality control mechanisms relies on the core features in this Astro-WISE Environment (AWE): an object-oriented framework, full data lineage, and both forward and backward chaining. Quality control information can be accessed via the command-line awe-prompt and t…
We describe the SpArc science gateway for spectral data obtained during the period from 1975 through 1995 at the Kitt Peak National Observatory using the Fourier Transform Spectrometer (FTS) in operation at the Mayall 4-m telescope. SpArc is hosted by Indiana University Bloomington and is available for public access. The archive includes nearly 10,000 individual spectra of more than 800 different astronomical sources including stars, nebulae, galaxies, and Solar System objects. We briefly describe the FTS instrument itself, and summarize the conversion of the original interferograms into spect…
Although the Gaia catalogue on its own will be a very powerful tool, it is the combination of this highly accurate archive with other archives that will truly open up amazing possibilities for astronomical research. The advanced interoperation of archives is based on cross-matching, leaving the user with the feeling of working with one single data archive. The data retrieval should work not only across data archives, but also across wavelength domains. The first step for seamless data access is the computation of the cross-match between Gaia and external surveys. The matching of astronomical c…
ESASky is a science-driven discovery portal to explore the multi-wavelength sky and visualise and access multiple astronomical archive holdings. The tool is a web application that requires no prior knowledge of any of the missions involved and gives users world-wide simplified access to the highest-level science data products from multiple astronomical space-based astronomy missions plus a number of ESA source catalogues. The first public release of ESASky features interfaces for the visualisation of the sky in multiple wavelengths, the visualisation of query results summaries, and the visuali…
The next decade will feature a growing number of massive ground-based photometric, spectroscopic, and time-domain surveys, including those produced by DECam, DESI, and LSST. The NOAO Data Lab was launched in 2017 to enable efficient exploration and analysis of large surveys, with particular focus on the petabyte-scale holdings of the NOAO Archive and their associated catalogs. The Data Lab mission and future development align well with two of the NSF's Big Ideas, namely Harnessing Data for 21st Century Science and Engineering and as part of a network to contribute to Windows on the Universe: T…
The Atacama Large Millimeter/submillimeter Array (ALMA), an international partnership of Europe, North America and East Asia in cooperation with the Republic of Chile, is the largest astronomical project in existence. While ALMA's capabilities are ramping up, Early Science observations have started. The ALMA Archive is at the center of the operations of the telescope array and is designed to manage the 200 TB of data that will be taken each year, once the observatory is in full operations. We briefly describe design principles. The second part of this paper focuses on how astronomy is likely t…
We present a bibliographic analysis of Chandra, Hubble, and Spitzer publications. We find (a) archival data are used in >60% of the publication output and (b) archives for these missions enable a much broader set of institutions and countries to scientifically use data from these missions. Specifically, we find that authors from institutions that have published few papers from a given mission publish 2/3 archival publications, while those with many publications typically have 1/3 archival publications. We also show that countries with lower GDP per capita overwhelmingly produce archival public…
With SKA precursor and pathfinder operations in full swing, radio and (sub-)mm astronomy is entering the era of super big data. The big questions is how to make (sub-)mm and radio data available to the astronomical community, preferably using FAIR (findable, accessible, interoperable and re-useable; Wilkinson et al. 2016) principles. There are already a lot of efforts going on around the globe: facilities such as ALMA, LOFAR, MWA, NRAO and ASKAP are already publishing much of their imaging data in the form of "science ready" products, SKA regional centres are being formed and a radio astronomy…
astroquery is a collection of tools for requesting data from databases hosted on remote servers with interfaces exposed on the internet, including those with web pages but without formal application program interfaces (APIs). These tools are built on the Python requests package, which is used to make HTTP requests, and astropy, which provides most of the data parsing functionality. astroquery modules generally attempt to replicate the web page interface provided by a given service as closely as possible, making the transition from browser-based to command-line interaction easy. astroquery has …
Besides new observations, mining old photographic plates and CCD image archives represents an opportunity to recover and secure newly discovered asteroids, also to improve the orbits of Near Earth Asteroids (NEAs), Potentially Hazardous Asteroids (PHAs) and Virtual Impactors (VIs). These are the main research aims of the EURONEAR network. As stated by the IAU, the vast collection of image archives stored worldwide is still insufficiently explored, and could be mined for known NEAs and other asteroids appearing occasionally in their fields. This data mining could be eased using a server to sear…
Commodity cloud computing, as provided by commercial vendors such as Amazon, Google, and Microsoft, has revolutionized computing in many sectors. With the advent of a new class of big data, public access astronomical facility such as LSST, DKIST, and WFIRST, there exists a real opportunity to combine these missions with cloud computing platforms and fundamentally change the way astronomical data is collected, processed, archived, and curated. Making these changes in a cross-mission, coordinated way can provide unprecedented economies of scale in personnel, data collection and management, archi…
Scientific exploitation of the ever increasing volumes of astronomical data requires efficient and practical methods for data access, visualisation, and analysis. Hierarchical sky tessellation techniques enable a multi-resolution approach to organising data on angular scales from the full sky down to the individual image pixels. Aims. We aim to show that the Hierarchical progressive survey (HiPS) scheme for describing astronomical images, source catalogues, and three-dimensional data cubes is a practical solution to managing large volumes of heterogeneous data and that it enables a new level o…
We present here a provenance management system adapted to astronomical projects needs. We collected use cases from various astronomy projects and defined a data model in the ecosystem developed by the IVOA (International Virtual Observatory Alliance). From those use cases, we observed that some projects already have data collections generated and archived, from which the provenance has to be extracted (provenance "on top"), and some projects are building complex pipelines that automatically capture provenance information during the data processing (capture "inside"). Different tools and protot…
Astronomical datasets are growing in size and diversity, posing severe technical problems. At the same time scientific goals increasingly require the analysis of very large amounts of data, and data from multiple archives. The Virtual Observatory (VO) initiative aims to make multiwavelength science and large database science as seamless as possible. It can be seen as the latest stage of a long term trend towards standardisation and collectivisation in astronomy. Within this inevitable trend, we can avoid the high energy style of building large fixed hierarchical teams, and keep the individuali…
Traditional analysis techniques may not be sufficient for astronomers to make the best use of the data sets that current and future instruments, such as the Square Kilometre Array and its Pathfinders, will produce. By utilizing the incredible pattern-recognition ability of the human mind, scientific visualization provides an excellent opportunity for astronomers to gain valuable new insight and understanding of their data, particularly when used interactively in 3D. The goal of our work is to establish the feasibility of a real-time 3D monitoring system for data going into the Australian SKA P…
In 2020, ~60PB of archived data will be accessible to the astronomers. But to analyze such a paramount data will be a challenging task. This is basically due to the computational model used to download the data from complex geographically distributed archives to a central site and then analyzing it in the local systems. Because the data has to be downloaded to the central site, the network BW limitation will be a hindrance for the scientific discoveries. Also analyzing this PB-scale on local machines in a centralized manner is challenging. In this virtual observatory is a step towards this pro…
The Zwicky Transient Facility is a new robotic-observing program, in which a newly engineered 600-MP digital camera with a pioneeringly large field of view, 47~square degrees, will be installed into the 48-inch Samuel Oschin Telescope at the Palomar Observatory. The camera will generate $\sim 1$~petabyte of raw image data over three years of operations. In parallel related work, new hardware and software systems are being developed to process these data in real time and build a long-term archive for the processed products. The first public release of archived products is planned for early 2019…
More than 20 years after the end of NASA's Compton Gamma-Ray Observatory mission, the data collected by its Imaging Compton Telescope (COMPTEL) still provide the most comprehensive and deepest view of our Universe in MeV gamma rays. While most of the COMPTEL data are archived at NASA's High Energy Astrophysics Science Archive Research Center (HEASARC), the absence of any publicly available software for their analysis means the data cannot benefit from the scientific advances made in the field of gamma-ray astronomy at higher energies. To make this unique treasure again accessible for science, …
AstroCloud is a cyber-Infrastructure for Astronomy Research initiated by Chinese Virtual Observatory (China-VO) under funding support from NDRC (National Development and Reform commission) and CAS (Chinese Academy of Sciences){\url{http://astrocloud.china-vo.org}}\citep{O8-5_Cui_adassxxiv}. To archive the astronomical data in China, we present the implementation of the astronomical data archiving system (ADAS). Data archiving and quality control are the infrastructure for the AstroCloud. Throughout the data of the entire life cycle, data archiving system standardized data, transferring data, l…
Intelligent scheduling of the sequence of scientific exposures taken at ground-based astronomical observatories is massively challenging. Observing time is over-subscribed and atmospheric conditions are constantly changing. We propose to guide observatory scheduling using machine learning. Leveraging a 15-year archive of exposures, environmental, and operating conditions logged by the Canada-France-Hawaii Telescope, we construct a probabilistic data-driven model that accurately predicts image quality. We demonstrate that, by optimizing the opening and closing of twelve vents placed on the dome…
The next generation of telescopes and instruments are facilitating our understanding of the Universe by producing data at a pace that beats all projections, and astronomers today are left in the face of an avalanche of data like never before. In order to cope with this problem and come up with a reliable and innovative solution, Data Centers were created in various locations and the concept of Virtual Observatories elaborated. Based at the National Taiwan Normal University, the Taiwan Extragalactic Astronomical Data Center plan to join in global efforts by proposing 1Pb of data storage dedicat…
Performing astronomical data analysis using only personal computers is becoming impractical for the very large data sets produced nowadays. As analysis is not a task that can be automatized to its full extent, the idea of moving processing where the data is located means also moving the whole scientific process towards the archives and data centers. Using Jupyter Notebooks as a remote service is a recent trend in data analysis that aims to deal with this problem, but harnessing the infrastructure to serve the astronomer without increasing the complexity of the service is a challenge. In this p…
The availability of vast quantities of data through electronic archives has transformed astronomical research. It has also enabled the creation of new products, models and simulations, often from distributed input data and models, that are themselves made electronically available. These products will only provide maximal long-term value to astronomers when accompanied by records of their provenance; that is, records of the data and processes used in the creation of such products. We use the creation of image mosaics with the Montage grid-enabled mosaic engine to emphasize the necessity of prov…
We present solar full-disk observations, which were recorded at the Einstein Tower during the years 1943 - 1991 (Solar Cycles 18 - 22). High-school students from Potsdam and Berlin digitized more than 3500 full-disk images during two- to three-week internships at AIP. The digital images cover a 15 cm $\times$ 15 cm region on photographic plates, which were scanned at a resolution of 7086 $\times$ 7086 pixels. The raw data are monochromatic 8-bit images in the Tagged Image File Format (TIFF). These images were calibrated and saved with improved photometric precision as 16-bit images with 2048 $…
This chapter describes how astronomical imaging survey data have become a vital part of modern astronomy, how these data are archived and then served to the astronomical community through on-line data access portals. The Virtual Observatory, now under development, aims to make all these data accessible through a uniform set of interfaces. This chapter also describes the scientific need for one common image processing task, that of composing individual images into large scale mosaics and introduces Montage as a tool for this task. Montage, as distributed, can be used in four ways: as a single t…
The field of gamma-ray astronomy has seen important progress during the last decade, yet there exists so far no common software framework for the scientific analysis of gamma-ray telescope data. We propose to fill this gap by means of the GammaLib software, a generic library that we have developed to support the analysis of gamma-ray event data. GammaLib has been written in C++ and all functionality is available in Python through an extension module. On top of this framework we have developed the ctools software package, a suite of software tools that enables building of flexible workflows for…
The virtual observatory (VO) is a collection of interoperable data archives, tools and applications that together form an environment in which original astronomical research can be carried out. The VO is opening up new ways of exploiting the huge amount of data provided by the ever-growing number of ground-based and space facilities, as well as by computer simulations. This presentation summarises a variety of scientific results spanning various fields of astronomy, obtained thanks to the VO, after highlighting its structure, infrastructure and various capabilities.
Observatories are complex scientific and technical institutions serving diverse users and purposes. Their telescopes, instruments, software, and human resources engage in interwoven workflows over a broad range of timescales. These workflows have been tuned to be responsive to concepts of observatory operations that were applicable when various assets were commissioned, years or decades in the past. The astronomical community is entering an era of rapid change increasingly characterized by large time domain surveys, robotic telescopes and automated infrastructures, and - most significantly - o…
