Data Provenance: Use Cases for the ESO archive, and Interactions with the Virtual Observatory
J. D. Santander-Vela, A. Delgado, N. Delmotte, M. Vuong
备注：
4 pages, 2 figures, to appear in the Proceedings of the Astronomical Data Analysis Software and Systems XIX meeting (as P64), in the ASP Conference Series
In the Virtual Observatory era, where we intend to expose scientists (or software agents on their behalf) to a stream of observations from all existing facilities, the ability to access and to further interpret the origin, relationships, and processing steps on archived astronomical assets (their Provenance) is a requirement for proper observation selection, and quality assessment. In this article we present the different use cases Data Provenance is needed for, the challenges inherent to building such a system for the ESO archive, and their link with ongoing work in the International Virtual Observatory Alliance (IVOA).
The ESO Science Archive is the collection and access point of the data generated at ESO's La Silla Paranal Observatory, both raw and processed. It is a major contributor to ESO's science output, being used in about 4 out of 10 refereed articles with ESO data. In this paper, which is presented on behalf of the operations and development teams, we review its contents, policies, us interfaces and impact.
We describe the VISTA Science Archive (VSA) and its first public release of data from five of the six VISTA Public Surveys. The VSA exists to support the VISTA Surveys through their lifecycle: the VISTA Public Survey consortia can use it during their quality control assessment of survey data products before submission to the ESO Science Archive Facility (ESO SAF); it supports their exploitation of survey data prior to its publication through the ESO SAF; and, subsequently, it provides the wider community with survey science exploitation tools that complement the data product repository functio…
The ESO/MPG WFI and the INT WFC wide field archives comprising 330,000 images were mined to search for serendipitous encounters of known Near Earth Asteroids (NEAs) and Potentially Hazardous Asteroids (PHAs). A total of 152 asteroids (44 PHAs and 108 other NEAs) were identified using the PRECOVERY software, their astrometry being measured on 761 images and sent to the Minor Planet Centre. Both recoveries and precoveries were reported, including prolonged orbital arcs for 18 precovered objects and 10 recoveries. We analyze all new opposition data by comparing the orbits fitted before and after …
The IVOA Provenance Data model defines entities, agents and activities as container classes to describe the provenance of datasets, with the executed tasks and responsibilities attached to agents. It also provides a set of classes to describe the activities type and their configuration template, as well as the configuration applied effectively during the execution of a task. Here we highlight lessons learned in the implementation of the CDS ProvHiPS service distributing provenance metadata for the HST HiPS data collections, and for the HST archive original images used to produce the HiPS tiles…
The Murchison Widefield Array (MWA) is a next-generation radio telescope, generating visibility data products continuously at about 400 MB/s. Efficiently managing and archiving this data is a challenge. The MWA Archive consists of dataflows and storage sub-systems distributed across three tiers. At its core is the open source software - the Next-Generation Archive System (NGAS) - that was initially developed in ESO. However, to meet the MWA data challenge, we have tailored and optimised NGAS to achieve high-throughput data ingestion, efficient dataflow management, multi-tiered data storage and…
Users of the La Silla Paranal Observatory have to rely on the ESO Exposure Time Calculator (ETC) to prepare their observations. A project has been started at ESO to modernise the ETC, based on a python backend and an angular-based front-end. The ETC 2.0 will have a programmatic interface to enable the results to be included in an automated quality control loop and to communicate with the Phase 1 proposal preparation and the Phase 2 observation preparation tools, the ESO science archive, as well as with scripts runs by external users or instruments. The first version of an ETC 2.0 has been rele…
We present here a provenance management system adapted to astronomical projects needs. We collected use cases from various astronomy projects and defined a data model in the ecosystem developed by the IVOA (International Virtual Observatory Alliance). From those use cases, we observed that some projects already have data collections generated and archived, from which the provenance has to be extracted (provenance "on top"), and some projects are building complex pipelines that automatically capture provenance information during the data processing (capture "inside"). Different tools and protot…
The availability of vast quantities of data through electronic archives has transformed astronomical research. It has also enabled the creation of new products, models and simulations, often from distributed input data and models, that are themselves made electronically available. These products will only provide maximal long-term value to astronomers when accompanied by records of their provenance; that is, records of the data and processes used in the creation of such products. We use the creation of image mosaics with the Montage grid-enabled mosaic engine to emphasize the necessity of prov…
The Additional Representative Images for Legacy (ARI-L) project is a European Development project for ALMA Upgrade approved by the Joint ALMA Observatory (JAO) and the European Southern Observatory (ESO), started in June 2019. It aims to increase the legacy value of the ALMA Science Archive (ASA) by bringing the reduction level of ALMA data from Cycles 2-4 close to that of data from more recent Cycles processed for imaging with the ALMA Pipeline. As of mid-2021 more than 150000 images have been returned to the ASA for public use. At its completion in 2022, the project will have provided enhanc…
ESO's two FOcal Reducer and low dispersion Spectrographs (FORS) are the primary optical imaging instruments for the VLT. They are not direct-imaging instruments, as there are several optical elements in the light path. In particular, both instruments are attached to a field rotator. Obtaining truly photometric data with such instruments present a significant challenge. In this paper, we investigate in detail twilight flats taken with the FORS instruments. We find that a large fraction of the structure seen in these flatfields rotates with the field rotator. We discuss in detail the methods we …
The world astronomical image archives represent huge opportunities to time-domain astronomy sciences and other hot topics such as space defense, and astronomical observatories should improve this wealth and make it more accessible in the big data era. In 2010 we introduced the Mega-Archive database and the Mega-Precovery server for data mining images containing Solar system bodies, with focus on near Earth asteroids (NEAs). This paper presents the improvements and introduces some new related data mining tools developed during the last five years. Currently, the Mega-Archive has indexed 15 mill…
The Gaia-ESO Public Spectroscopic Survey is an ambitious project designed to obtain astrophysical parameters and elemental abundances for 100,000 stars, including large representative samples of the stellar populations in the Galaxy, and a well-defined sample of 60 (plus 20 archive) open clusters. We provide internally consistent results calibrated on benchmark stars and star clusters, extending across a very wide range of abundances and ages. This provides a legacy data set of intrinsic value, and equally a large wide-ranging dataset that is of value for homogenisation of other and future ste…
The Son-Of-XShooter (SOXS) is a single object spectrograph (UV-VIS & NIR) and acquisition camera scheduled to be mounted on the ESO 3.58-m New Technology Telescope at the La Silla Observatory. Although the underlying data reduction processes to convert raw detector data to fully-reduced science ready data are complex and multi-stepped, we have designed the SOXS Data Reduction pipeline with the core aims of providing end-users with a simple-to-use, well-documented command-line interface while also allowing the pipeline to be run in a fully automated state; streaming reduced data into the ESO Sc…
The Public European Southern Observatory Spectroscopic Survey of Transient Objects (PESSTO) began as a public spectroscopic survey in April 2012. We describe the data reduction strategy and data products which are publicly available through the ESO archive as the Spectroscopic Survey Data Release 1 (SSDR1). PESSTO uses the New Technology Telescope with EFOSC2 and SOFI to provide optical and NIR spectroscopy and imaging. We target supernovae and optical transients brighter than 20.5mag for classification. Science targets are then selected for follow-up based on the PESSTO science goal of extend…
Telescope bibliographies are integral parts of observing facilities. They are used to associate the published literature with archived observational data, to measure an observatory's scientific output through publication and citation statistics, and to define guidelines for future observing strategies. The ESO and NRAO librarians as well as NAOJ jointly maintain the ALMA (Atacama Large Millimeter/submillimeter Array) bibliography, a database of refereed papers that use ALMA data. In this paper, we illustrate how relevant articles are identified, which procedures are used to tag entries in the …
We present the PHANGS-MUSE survey, a programme using the MUSE IFS at the ESO VLT to map 19 massive $(9.4 < \log(M_{*}/M_\odot) < 11.0)$ nearby (D < 20 Mpc) star-forming disc galaxies. The survey consists of 168 MUSE pointings (1'x1' each), a total of nearly 15 Million spectra, covering ~1.5 Million independent spectra. PHANGS-MUSE provides the first IFS view of star formation across different local environments (including galaxy centres, bars, spiral arms) in external galaxies at a median resolution of 50~pc, better than the mean inter-cloud distance in the ionised interstellar medium. This `c…
