The Montage image mosaic engine has found wide applicability in astronomy research, integration into processing environments, and is an examplar application for the development of advanced cyber-infrastructure. It is written in C to provide performance and portability. Linking C/C++ libraries to the Python kernel at run time as binary extensions allows them to run under Python at compiled speeds and enables users to take advantage of all the functionality in Python. We have built Python binary extensions of the 59 ANSI-C modules that make up version 5 of the Montage toolkit. This has involved a turning the code into a C library, with driver code fully separated to reproduce the calling sequence of the command-line tools; and then adding Python and C linkage code with the Cython library, which acts as a bridge between general C libraries and the Python interface. We will demonstrate how to use these Python binary extensions to perform image processing, including reprojecting and resampling images, rectifying background emission to a common level, creation of image mosaics that preserve the calibration and astrometric fidelity of the input images, creating visualizations with an adaptive stretch algorithm, processing HEALPix images, and analyzing and managing image metadata.
The scientific computing landscape has evolved dramatically in the past few years, with new schemes for organizing and storing data that reflect the growth in size and complexity of astronomical data sets. In response to this changing landscape, we are, over the next two years, deploying the next generation of the Montage toolkit ([ascl:1010.036]). The first release (October 2015) supports multi-dimensional data sets ("data cubes"), and insertion of XMP/AVM tags that allows images to "drop-in" to the WWT. The same release offers a beta-version of web-based interactive visualization of images; …
This paper describes how we have sustained the Montage image mosaic engine (http://montage.ipac.caltech.edu) first released in 2002, to support the ever-growing scale and complexity of modern data sets. The key to its longevity has been its design as a toolkit written in ANSI-C, with each tool performing one distinct task, for easy integration into scripts, pipelines and workflows. The same code base now supports Windows, JavaScript and Python by taking advantage of recent advances in compilers. The design has led to applicability of Montage far beyond what was anticipated when Montage was fir…
Image processing at scale is a powerful tool for creating new data sets and integrating them with existing data sets and performing analysis and quality assurance investigations. Workflow managers offer advantages in this type of processing, which involves multiple data access and processing steps. Generally, they enable automation of the workflow by locating data and resources, recovery from failures, and monitoring of performance. In this focus demo we demonstrate how the Pegasus Workflow Manager Python API manages image processing to create mosaics with the Montage Image Mosaic engine. Sinc…
We describe a case study to use the Montage image mosaic engine to create maps of the ALLWISE image data set in the Hierarchical Progressive Survey (HiPS) sky-tesselation scheme. Our approach demonstrates that Montage reveals the science content of infrared images in greater detail than has hitherto been possible in HiPS maps. The approach exploits two unique (to our knowledge) characteristics of the Montage image mosaic engine: background modeling to rectify the time variable image backgrounds to common levels; and an adaptive image stretch to present images for visualization. The creation of…
The Montage Image Mosaic Engine was designed as a scalable toolkit, written in C for performance and portability across *nix platforms, that assembles FITS images into mosaics. The code is freely available and has been widely used in the astronomy and IT communities for research, product generation and for developing next-generation cyber-infrastructure. Recently, it has begun to finding applicability in the field of visualization. This has come about because the toolkit design allows easy integration into scalable systems that process data for subsequent visualization in a browser or client. …
We have used the Montage image mosaic engine to investigate the cost and performance of processing images on the Amazon EC2 cloud, and to inform the requirements that higher-level products impose on provenance management technologies. We will present a detailed comparison of the performance of Montage on the cloud and on the Abe high performance cluster at the National Center for Supercomputing Applications (NCSA). Because Montage generates many intermediate products, we have used it to understand the science requirements that higher-level products impose on provenance management technologies.…
This chapter describes how astronomical imaging survey data have become a vital part of modern astronomy, how these data are archived and then served to the astronomical community through on-line data access portals. The Virtual Observatory, now under development, aims to make all these data accessible through a uniform set of interfaces. This chapter also describes the scientific need for one common image processing task, that of composing individual images into large scale mosaics and introduces Montage as a tool for this task. Montage, as distributed, can be used in four ways: as a single t…
We present the open-source image processing software package PySAP (Python Sparse data Analysis Package) developed for the COmpressed Sensing for Magnetic resonance Imaging and Cosmology (COSMIC) project. This package provides a set of flexible tools that can be applied to a variety of compressed sensing and image reconstruction problems in various research domains. In particular, PySAP offers fast wavelet transforms and a range of integrated optimisation algorithms. In this paper we present the features available in PySAP and provide practical demonstrations on astrophysical and magnetic reso…
High performance computing has been used in various fields of astrophysical research. But most of it is implemented on massively parallel systems (supercomputers) or graphical processing unit clusters. With the advent of multicore processors in the last decade, many serial software codes have been re-implemented in parallel mode to utilize the full potential of these processors. In this paper, we propose parallel processing recipes for multicore machines for astronomical data processing. The target audience are astronomers who are using Python as their preferred scripting language and who may …
MESAlab: A Python pipeline for MESA grid processing with integrated GYRE/MESA-RSP execution
Konkoly Observatory, HUN-REN CSFK, MTA Centre of Excellence（孔科利天文台，HUN-REN 匈牙利科学院研究中心，MTA 卓越中心）；MTA–HUN-REN CSFK Lendület ”Momentum” Stellar Pulsation Research Group（MTA-HUN-REN 匈牙利科学院研究中心 Lendulet “动量”恒星脉动研究组）
To reduce and analyze astronomical images, astronomers can rely on a wide range of libraries providing low-level implementations of legacy algorithms. However, combining these routines into robust and functional pipelines requires a major effort which often ends up in instrument-specific and poorly maintainable tools, yielding products that suffer from a low-level of reproducibility and portability. In this context, we present prose, a Python framework to build modular and maintainable image processing pipelines. Built for astronomy, it is instrument-agnostic and allows the construction of pip…
Radio astronomy observatories with high throughput back end instruments require real-time data processing. While computing hardware continues to advance rapidly, development of real-time processing pipelines remains difficult and time-consuming, which can limit scientific productivity. Motivated by this, we have developed Bifrost: an open-source software framework for rapid pipeline development. Bifrost combines a high-level Python interface with highly efficient reconfigurable data transport and a library of computing blocks for CPU and GPU processing. The framework is generalizable, but init…
As modern scientific simulations grow ever more in size and complexity, even their analysis and post-processing becomes increasingly demanding, calling for the use of HPC resources and methods. yt is a parallel, open source post-processing python package for numerical simulations in astrophysics, made popular by its cross-format compatibility, its active community of developers and its integration with several other professional Python instruments. The Intel Distribution for Python enhances yt's performance and parallel scalability, through the optimization of lower-level libraries Numpy and S…
We announce ChromaStarPy, an integrated general stellar atmospheric modeling and spectrum synthesis code written entirely in python V. 3. ChromaStarPy is a direct port of the ChromaStarServer (CSServ) Java modeling code described in earlier papers in this series, and many of the associated JavaScript (JS) post-processing procedures have been ported and incorporated into CSPy so that students have access to ready-made "data products". A python integrated development environment (IDE) allows a student in a more advanced course to experiment with the code and to graphically visualize intermediate…
PESummary is a Python software package for processing and visualising data from any parameter estimation code. The easy to use Python executable scripts and extensive online documentation has resulted in PESummary becoming a key component in the international gravitational-wave analysis toolkit. PESummary has been developed to be more than just a post-processing tool with all outputs fully self-contained. PESummary has become central to making gravitational-wave inference analysis open and easily reproducible.
We introduce AXS (Astronomy eXtensions for Spark), a scalable open-source astronomical data analysis framework built on Apache Spark, a widely used industry-standard engine for big data processing. Building on capabilities present in Spark, AXS aims to enable querying and analyzing almost arbitrarily large astronomical catalogs using familiar Python/AstroPy concepts, DataFrame APIs, and SQL statements. We achieve this by i) adding support to Spark for efficient on-line positional cross-matching and ii) supplying a Python library supporting commonly-used operations for astronomical data analysi…
We present the Vortex Image Processing (VIP) library, a python package dedicated to astronomical high-contrast imaging. Our package relies on the extensive python stack of scientific libraries and aims to provide a flexible framework for high-contrast data and image processing. In this paper, we describe the capabilities of VIP related to processing image sequences acquired using the angular differential imaging (ADI) observing technique. VIP implements functionalities for building high-contrast data processing pipelines, encompass- ing pre- and post-processing algorithms, potential sources po…
We present PyWiFeS, a new Python-based data reduction pipeline for the Wide Field Spectrograph (WiFeS). PyWiFeS consists of a series of core data processing routines built on standard scientific Python packages commonly used in astronomical applications. Included in PyWiFeS is an implementation of a new global optical model of the spectrograph which provides wavelengths solutions accurate to $\sim$0.05 Å (RMS) across the entire detector. The core PyWiFeS package is designed to be scriptable to enable batch processing of large quantities of data, and we present a default format for handling of …
The pyFAI package has been designed to reduce X-ray diffraction images into powder diffraction curves to be further processed by scientists. This contribution describes how to convert an image into a radial profile using the Numpy package, how the process was accelerated using Cython. The algorithm was parallelised, needing a complete re-design to benefit from massively parallel devices like graphical processing units or accelerators like the Intel Xeon Phi using the PyOpenCL library.
anesthetic is a Python package for processing nested sampling runs, and will be useful for any scientist or statistician who uses nested sampling software. anesthetic unifies many existing tools and techniques in an extensible framework that is intuitive for users familiar with the standard Python packages, namely NumPy, SciPy, Matplotlib and pandas.
University of Potsdam（波茨坦大学）；Humboldt University of Berlin（柏林洪堡大学）
HexagDLy is a Python-library extending the PyTorch deep learning framework with convolution and pooling operations on hexagonal grids. It aims to ease the access to convolutional neural networks for applications that rely on hexagonally sampled data as, for example, commonly found in ground-based astroparticle physics experiments.
There are a number of different phenomena in the early universe that have to be studied numerically with lattice simulations. This paper presents a graphics processing unit (GPU) accelerated Python program called PyCOOL that solves the evolution of scalar fields in a lattice with very precise symplectic integrators. The program has been written with the intention to hit a sweet spot of speed, accuracy and user friendliness. This has been achieved by using the Python language with the PyCUDA interface to make a program that is easy to adapt to different scalar field models. In this paper we der…
We present Artificial Stellar Populations (ArtPop), an open-source Python package for synthesizing stellar populations and generating artificial images of fully populated stellar systems. The code is designed to be intuitive to use and as modular as possible, making it possible to use each of its functionalities independently or together. ArtPop has a wide range of scientific and pedagogical use cases, including the measurement of detection efficiencies in current and future imaging surveys, the calculation of integrated stellar population parameters, quantitative comparisons of isochrone mode…
