Enhancing interoperable data access to radio data has become a science priority within the International Virtual Observatory Alliance (IVOA). This lead to the foundation of the IVOA Radio astronomy Interest Group. Several radio astronomers and project scientists enrolled in various projects (NRAO, ASKAP, LOFAR, JIVE, ALMA, SKA, INAF, NenuFAR, etc.) have joined. Together they are paving the way to a better integration of their services in the virtual observatory (VO) infrastructure and propose extension of IVOA standards to help achieving this goal. Calibrated radio datasets such as cubes, images, spectra and time series can already be searched and retrieved using the ObsCore/ObsTAP specification defined in the IVOA, or by data product-specific services like SIAv2, SODA, SSA and ConeSearch. However, properties of radio visibility data are not fully implemented in the VO landscape yet. We need specific features to refine data discovery and selection that are adapted to radio astronomers' need. In this context the VO team at the Centre de Données astronomiques de Strasbourg (CDS) proposes to consider the ObsCore/ObsTAP specification and to establish cross-walks between the ObsCore and the existing Measurement Set (MS) metadata profile for data discovery of radio visibility data (VD). In order to account for the difference in granularity between radio VD datasets and science-ready datasets of the VO, the approach splits a MS data file into a list of datasets served by an ObsTAP service, thus enabling coarse grain discovery in the multi-wavelength context. Radio specific metadata such as number of antennae, frequency ranges, $uv$ plane coverage plots, frequency-phase and frequency-amplitude plots, primary and synthesized beams are also provided either by adding column metadata or by using the DataLink technique.
This document describes the Astronomical Data Query Language (ADQL). ADQL has been developed based on SQL92. This document describes the subset of the SQL grammar supported by ADQL. Special restrictions and extensions to SQL92 have been defined in order to support generic and astronomy specific operations.
SAMP is a messaging protocol that enables astronomy software tools to interoperate and communicate. IVOA members have recognised that building a monolithic tool that attempts to fulfil all the requirements of all users is impractical, and it is a better use of our limited resources to enable individual tools to work together better. One element of this is defining common file formats for the exchange of data between different applications. Another important component is a messaging system that enables the applications to share data and take advantage of each other's functionality. SAMP builds …
Handling, processing and archiving the huge amount of data produced by the new generation of experiments and instruments in Astronomy and Astrophysics are among the more exciting challenges to address in designing the future data management infrastructures and computing services. We investigated the feasibility of a data management and computation infrastructure, available world-wide, with the aim of merging the FAIR (Findable, Accessible, Interoperable, Reusable) data management provided by IVOA standards with the efficiency and reliability of a cloud approach. Our work involved the Canadian …
Data Models are an essential part of automatic data processing, but even more so when trying to tie together data coming from many different data sources, as is the case for the International Virtual Observatory. In this talk we will review the different data models used in the IVOA, which parts of that Data Modelling work are still incomplete, specially in radio wavelengths, and the work the AMIGA group has done within the IVOA Data Modelling Working Group to overcome those shortcomings both in missing data models and support for Radio Astronomy.
In the context of astronomy projects, scientists have been confronted with the problem of describing in a standardized way how their data have been produced. As presented in a talk at last year's ADASS, the International Virtual Observatory Alliance (IVOA) is working on the definition of a Provenance Data Model, compatible with the W3C PROV model, which shall describe how provenance metadata can be modeled, stored and exchanged in astronomy. In this poster, we present the current status of our developments of libraries and tools, mainly open source, which implement the IVOA Provenance Data Mod…
We present the last developments on the IVOA Provenance data model, mainly based on the W3C PROV concept. In the context of the Cherenkov astronomy, the data processing stages imply both assumptions and comparison to dedicated simulations. As a consequence, Provenance information is crucial to the end user in order to interpret the high level data products. The Cherenkov Telescope Array (CTA), currently in preparation, is thus a perfect test case for the development of an IVOA standard on Provenance information. We describe general use-cases for the computational Provenance in the CTA producti…
We are in the era of the Big Data. In Astronomy and Astrophysics, the massive amounts of data generated are, as of today, in the Peta-scale if not already in the Exa-scale. In the near future, we will see the data collected size and complexity grow further constantly, setting new challenges for data processing, reduction and analysis. This will pose new needs in terms of software and hardware solutions but also in terms of new models for resource management, access and sharing. In Astronomy and Astrophysics, in the environment of the International Virtual Observatory Alliance (IVOA), a big wor…
With the success and widespread of the IVOA Table Access Protocol (1) for discovering and querying tabular data in astronomy, more than one hundred of TAP services exposing altogether 22 thousands of tables are accessible from the IVOA Registries at the time of writing. Currently the TAP protocol presents table data and metadata via a {TAP\_SCHEMA} describing the served tables with their columns and possible joins between them. We explore here how to add an information layer, so that values within table columns can be gathered and used to populate instances of objects defined in a selected IVO…
The situation of data sharing in astronomy is positioned in the current general context of a political push towards, and rapid development of, scientific data sharing. Data is already one of the major infrastructures of astronomy, thanks to the data and service providers and to the International Virtual Observatory Alliance (IVOA). Other disciplines are moving on in the same direction. International organisations, in particular the Research Data Alliance (RDA), are developing building blocks and bridges to enable scientific data sharing across borders. The liaisons between RDA and astronomy, a…
Since its inception in the early 2000, the Virtual Observatory (VO), developed as a collaboration of many national and international projects, has become a major factor in the discovery and dissemination of astronomical information worldwide. The IVOA has been coordinating all these efforts worldwide to ensure a common VO framework that enables transparent access to and interoperability of astronomy resources (data and software) around the world. The VO is not a magic solution to all astronomy data management challenges but it does bring useful solutions in many areas borne out by the fact tha…
The IVOA works towards standardising interoperability and curation of data and service holdings of the global astrophysical community. Within the IVOA, the Data Access Layer (DAL) Working Group's goal is to provide technical standards for accessing data collections and catalogues; filtering data holdings based on their metadata; and retrieving the ones in scope, or operating on them. In recent years the DAL community has addressed the multi-dimensional and multi-wavelength scenarios, and kept core standards up-to-date. It has also tackled new topics such as the observation location and object …
Data access and interoperability module connects the observation proposals, data, virtual machines and software. According to the unique identifier of PI (principal investigator), an email address or an internal ID, data can be collected by PI's proposals, or by the search interfaces, e.g. conesearch. Files associated with the searched results could be easily transported to cloud storages, including the storage with virtual machines, or several commercial platforms like Dropbox. Benefitted from the standards of IVOA (International Observatories Alliance), VOTable formatted searching result cou…
Recently the International Virtual Observatory Alliance (IVOA) released a standard to structure provenance metadata, and several implementations are in development in order to capture, store, access and visualize the provenance of astronomy data products. This BoF will be focused on practical needs for provenance in astronomy. A growing number of projects express the requirement to propose FAIR data (Findable, Accessible, Interoperable and Reusable) and thus manage provenance information to ensure the quality, reliability and trustworthiness of this data. The concepts are in place, but now, ap…
With the implementation of many ambitious observation projects, including LAMOST, FAST, and Antarctic observatory at Doom A, observational astronomy in China is stepping into a brand new era with emerging data avalanche. In the era of e-Science, both these cutting-edge projects and traditional astronomy research need much more powerful data management, sharing and interoperability. Based on data-grid concept, taking advantages of the IVOA interoperability technologies, China-VO is developing a VO-driven astronomical data grid environment to enable multi-wavelength science and large database sc…
IAU Inter-Commission B2-C1-C2 WG Data-driven Astronomy Education and Public Outreach (DAEPO) was launched officially in April 2017. With the development of many mega-science astronomical projects, for example CTA, DESI, EUCLID, FAST, GAIA, JWST, LAMOST, LSST, SDSS, SKA, and large scale simulations, astronomy has become a Big Data science. Astronomical data is not only necessary resource for scientific research, but also very valuable resource for education and public outreach (EPO), especially in the era of Internet and Cloud Computing. IAU WG Data-driven Astronomy Education and Public Outreac…
A joint project between the Canadian Astronomy Data Center of the National Research Council Canada, and the italian Istituto Nazionale di Astrofisica-Osservatorio Astronomico di Trieste (INAF-OATs), partially funded by the EGI-Engage H2020 European Project, is devoted to deploy an integrated infrastructure, based on the International Virtual Observatory Alliance (IVOA) standards, to access and exploit astronomical data. Currently CADC-CANFAR provides scientists with an access, storage and computation facility, based on software libraries implementing a set of standards developed by the Interna…
The landscape of ground-based gamma-ray astronomy is drastically changing with the perspective of the Cherenkov Telescope Array (CTA) composed of more than 100 Cherenkov telescopes. For the first time in this energy domain, CTA will be operated as an observatory open to the astronomy community. In this context, a structured high level data model is being developed to describe a CTA observation. The data model includes different classes of metadata on the project definition, the configuration of the instrument, the ambient conditions, the data acquisition and the data processing. This last part…
With SKA precursor and pathfinder operations in full swing, radio and (sub-)mm astronomy is entering the era of super big data. The big questions is how to make (sub-)mm and radio data available to the astronomical community, preferably using FAIR (findable, accessible, interoperable and re-useable; Wilkinson et al. 2016) principles. There are already a lot of efforts going on around the globe: facilities such as ALMA, LOFAR, MWA, NRAO and ASKAP are already publishing much of their imaging data in the form of "science ready" products, SKA regional centres are being formed and a radio astronomy…
Closure phases and closure amplitudes have proven critical to modern radio interferometry due to their insensitivity to the uncertain station gains. We present the first set of closure quantities constructed from parallel-hand and cross-hand visibilities that are insensitive to both station gains and to polarimetric leakage. These complex "closure traces" are a natural extension of closure amplitudes and closure phases, are independent of all station-based linear corruptions of the polarized visibilities, and are complete in the sense that they contain all remaining information present in the …
Transient radio signals of astrophysical origin present an avenue for studying the dynamic universe. With the next generation of radio interferometers being planned and built, there is great potential for detecting and studying large samples of radio transients. Currently-used image-based techniques for detecting radio sources have not been demonstrated to be optimal, and there is a need for development of more sophisticated algorithms, and methodology for comparing different detection techniques. A visibility-space detector benefits from our good understanding of visibility-space noise proper…
We present RESOLVE, a new algorithm for radio aperture synthesis imaging of extended and diffuse emission in total intensity. The algorithm is derived using Bayesian statistical inference techniques, estimating the surface brightness in the sky assuming a priori log-normal statistics. RESOLVE not only estimates the measured sky brightness in total intensity, but also its spatial correlation structure, which is used to guide the algorithm to an optimal reconstruction of extended and diffuse sources. For a radio interferometer, it succeeds in deconvolving the effects of the instrumental point sp…
The Square Kilometre Array (SKA) is the largest radio interferometer under construction in the world. Its immense amount of visibility data poses a considerable challenge to the subsequent processing by the science data processor (SDP). Baseline dependent averaging (BDA), which reduces the amount of visibility data based on the baseline distribution of the radio interferometer, has become a focus of SKA SDP development. This paper developed and implemented a full-featured BDA module based on Radio Astronomy Simulation, Calibration and Imaging Library (RASCIL). Simulated observations were then …
We describe the process of stacking radio interferometry visibilities to form a deep composite image and its application to the observation of transient phenomena. We apply "visibility stacking" to 46 archival Very Large Array observations of nearby type Ia supernovae (SNeIa). This new approach provides an upper limit on the SNIa ensemble peak radio luminosity of 1.2x10^{25}erg/s/Hz at 5GHz, which is 5-10 times lower than previously measured. This luminosity implies an upper limit on the average companion stellar wind mass loss rate of 1.3x10^{-7}M_o/yr. This mass loss rate is consistent with …
Experience suggests that structural issues in how institutional Astrophysics approaches data-driven science and the development of discovery technology may be hampering the community's ability to respond effectively to a rapidly changing environment in which increasingly complex, heterogeneous datasets are challenging our existing information infrastructure and traditional approaches to analysis. We stand at the confluence of a new epoch of multimessenger science, remote co-location of data and processing power and new observing strategies based on miniaturized spacecraft. Significant effort w…
Options for Compression of radio interferometry data: lossy compression of visibilities and lossless compression of uv-visibility grids for the MHONGOOSE survey
International Centre for Radio Astronomy Research (ICRAR), University of Western Australia（西澳大利亚大学国际射电天文研究中心）；Australia Telescope National Facility, CSIRO Space & Astronomy（CSIRO空间与天文局澳大利亚望远镜国家设施）；Netherlands Institute for Radio Astronomy (ASTRON)（荷兰射电天文研究所）；Dept. of Astronomy, Univ. of Cape Town（开普敦大学天文学系）；Kapteyn Astronomical Institute, University of Groningen（格罗宁根大学卡普坦天文研究所）；Australian SKA Regional Centre (AusSRC), University of Western Australia（西澳大利亚大学澳大利亚SKA区域中心）
In the past two decades, a rebirth of interest in low-frequency radio astronomy for 21 cm tomography of the Epoch of Reionization, has given rise to a new class of radio interferometers with $N \gg 100$ antennas. The availability of low-noise receivers that do not require cryogenic cooling has driven down the cost of antennas, making it affordable to build sensitivity with numerous small antennas rather than large dish structures. However, the computational- and storage-costs of such radio arrays, determined by the $\mathcal{O}(N^2)$ scaling of visibility products required for calibration and …
We discuss current efforts behind the curation of observing proposals, archive bibliographies, and data links in the NASA Astrophysics Data System (ADS). The primary data in the ADS is the bibliographic content from scholarly articles in Astronomy and Physics, which ADS aggregates from publishers, arXiv and conference proceeding sites. This core bibliographic information is then further enriched by ADS via the generation of citations and usage data, and through the aggregation of external resources from astronomy data archives and libraries. Important sources of such additional information are…
The Square Kilometer Array (SKA) would be the world's largest radio telescope with eventually over a square kilometer of collecting area. However, there are enormous challenges in its data processing. The using of modern distributed computing techniques to solve the problem of massive data processing in SKA is one of the most important challenges. In this study, basing on the Dask distribution computational framework, and taking the visibility function integral processing as an example, we adopt a multi-level parallelism method to implement distributed averaging over time and channel. Dask Arr…
The SAO/NASA Astrophysics Data System (ADS) is an open access digital library portal for researchers in astronomy and physics, operated by the Smithsonian Astrophysical Observatory (SAO) under a NASA grant, successfully serving the professional science community for two decades. Currently there are about 55,000 frequent users (100+ queries per year), and up to 10 million infrequent users per year. Access by the general public now accounts for about half of all ADS use, demonstrating the vast reach of the content in our databases. The visibility and use of content in the ADS can be measured by …
The Europlanet-2020 programme, which ended on Aug 31st, 2019, included an activity called VESPA (Virtual European Solar and Planetary Access), which focused on adapting Virtual Observatory (VO) techniques to handle Planetary Science data. This paper describes some aspects of VESPA at the end of this 4-years development phase and at the onset of the newly selected Europlanet-2024 programme starting in 2020. The main objectives of VESPA are to facilitate searches both in big archives and in small databases, to enable data analysis by providing simple data access and online visualization function…
Hydrogen intensity mapping is a new field in astronomy that promises to make three-dimensional maps of the matter distribution of the Universe using the redshifted $21\,\textrm{cm}$ line of neutral hydrogen gas (HI). Several ongoing and upcoming radio interferometers, such as Tianlai, CHIME, HERA, HIRAX, etc. are using this technique. These instruments are designed to map large swaths of the sky by drift scanning over periods of many months. One of the challenges of the observations is that the daytime data is contaminated by strong radio signals from the Sun. In the case of Tianlai, this resu…
A Conditional Denoising Diffusion Probabilistic Model for Radio Interferometric Image Reconstruction
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong University of Science and Technology（香港科技大学）；Guangzhou University（广州大学）
IRIS: A Bayesian Approach for Image Reconstruction in Radio Interferometry with expressive Score-Based priors
Université de Montréal（蒙特利尔大学）；Mila - Quebec Artificial Intelligence Institute（米拉-魁北克人工智能研究所）；Ciela - Montreal Institute for Astrophysical Data Analysis and Machine Learning（CIELA-蒙特利尔天体物理数据分析与机器学习研究所）；University of Oxford（牛津大学）；Flatiron Institute - Center for Computational Astrophysics（弗拉蒂伦研究所-计算天体物理中心）；Perimeter Institute（圆周理论物理研究所）；McGill University（麦吉尔大学）；University of Manchester（曼彻斯特大学）；The Alan Turing Institute（阿兰·图灵研究所）
本文提出贝叶斯成像框架 IRIS，将星系光学图像训练的基于分数模型先验与 uv 空间高斯似然结合，用于 ALMA DSHARP 原行星盘可见度重建，并通过模拟覆盖率测试验证后验校准。
