The impulsive noise in astronomical images originates from various sources. It develops as a result of thermal generation in pixels, collision of cosmic rays with image sensor or may be induced by high readout voltage in Electron Multiplying CCD (EMCCD). It is usually efficiently removed by employing the dark frames or by averaging several exposures. Unfortunately, there are some circumstances, when either the observed objects or positions of impulsive pixels evolve and therefore each obtained image has to be filtered independently. In this article we present an overview of impulsive noise filtering methods and compare their efficiency for the purpose of astronomical image enhancement. The employed set of noise templates consists of dark frames obtained from CCD and EMCCD cameras working on ground and in space. The experiments conducted on synthetic and real images, allowed for drawing numerous conclusions about the usefulness of several filtering methods for various: (1) widths of stellar profiles, (2) signal to noise ratios, (3) noise distributions and (4) applied imaging techniques. The results of presented evaluation are especially valuable for selection of the most efficient filtering schema in astronomical image processing pipelines.
KMOS is a multi-object near-infrared integral field spectrometer with 24 deployable cryogenic pick-off arms. Inevitably, data processing is a complex task that requires careful calibration and quality control. In this paper we describe all the steps involved in producing science-quality data products from the raw observations. In particular, we focus on the following issues: (i) the calibration scheme which produces maps of the spatial and spectral locations of all illuminated pixels on the detectors; (ii) our concept of minimising the number of interpolations, to the limiting case of a single…
Data from complex modern astronomical instruments often consist of a large number of different science and calibration files, and their reduction requires a variety of software tools. The execution chain of the tools represents a complex workflow that needs to be tuned and supervised, often by individual researchers that are not necessarily experts for any specific instrument. The efficiency of data reduction can be improved by using automatic workflows to organise data and execute the sequence of data reduction steps. To realize such efficiency gains, we designed a system that allows intuitiv…
We present a new method of interpolation for the pixel brightness estimation in astronomical images. Our new method is simple and easily implementable. We show the comparison of this method with the widely used linear interpolation and other interpolation algorithms using one thousand astronomical images obtained from the Sloan Digital Sky Survey. The comparison shows that our method improves bad pixels brightness estimation with four times lower mean error than the presently most popular linear interpolation and has a better performance than any other examined method. The presented idea is fl…
phoptic -- a Python package for reducing astronomical images
School of Physics and Astronomy University of Southampton（南安普顿大学物理与天文学学院）；Department of Physics University of Warwick（华威大学物理系）；Fluid and Complex Systems Centre Coventry University（考文垂大学流体与复杂系统中心）；Instituto de Astronomía Universidad Nacional Autónoma de México（墨西哥国立自治大学天文研究所）；Universidad Nacional de Córdoba Observatorio Astronómico de Córdoba（科尔多瓦国立大学科尔多瓦天文台）；Consejo Nacional de Investigaciones Científicas y Téchnicas (CONICET)（国家科学研究委员会）；ComplyAdvantage；SUPA Physics and Astronomy University of St Andrews（圣安德鲁斯大学物理学与天文学）
Astronomical images are essential for exploring and understanding the universe. Optical telescopes capable of deep observations, such as the Hubble Space Telescope, are heavily oversubscribed in the Astronomical Community. Images also often contain additive noise, which makes de-noising a mandatory step in post-processing the data before further data analysis. In order to maximise the efficiency and information gain in the post-processing of astronomical imaging, we turn to machine learning. We propose Astro U-net, a convolutional neural network for image de-noising and enhancement. For a proo…
Spectroscopy is one of the most important tools that an astronomer has for studying the universe. This chapter begins by discussing the basics, including the different types of optical spectrographs, with extension to the ultraviolet and the near-infrared. Emphasis is given to the fundamentals of how spectrographs are used, and the trade-offs involved in designing an observational experiment. It then covers observing and reduction techniques, noting that some of the standard practices of flat-fielding often actually degrade the quality of the data rather than improve it. Although the focus is …
Scintillation noise due to the Earth's turbulent atmosphere can be a dominant noise source in high-precision astronomical photometry when observing bright targets from the ground. Here we describe the phenomenon of scintillation from its physical origins to its effect on photometry. We show that Young's (1967) scintillation-noise approximation used by many astronomers tends to underestimate the median scintillation noise at several major observatories around the world. We show that using median atmospheric optical turbulence profiles, which are now available for most sites, provides a better e…
Photometric surveys have provided incredible amounts of astronomical information in the form of images. However, astronomical images often contain artifacts that can critically hinder scientific analysis by misrepresenting intensities or contaminating catalogs as artificial objects. These affected pixels need to be masked and dealt with in any data reduction pipeline. In this paper, we present a flexible, iterative algorithm to recover (unmask) astronomical images where some pixels are lacking. We demonstrate the application of the method on some intensity calibration source images in CO Multi…
The Euclid satellite is an ESA mission scheduled for launch in September 2023. To optimally perform critical stages of the data reduction, such as object detection and morphology determination, a new and modern software package was required. We have developed SourceXtractor++ as open source software for detecting and measuring sources in astronomical images. It is a complete redesign of the original SExtractor, written mainly in C++. The package follows a modular approach and facilitates the analysis of multiple overlapping sources over many images with different pixel grids. SourceXtractor++ …
GPUPHOT: A Python Framework for High-Performance GPU-Accelerated Photometry and Distributed Astronomical Data Reduction
Light Bridges S.L.；Universidad de La Laguna (ULL)（拉帕尔马大学）；Instituto Tecnológico y de Energías Renovables (ITER)（技术与可再生能源研究所）；Instituto de Astrofísica de Canarias (IAC)（加那利天体物理研究所）
In this work, we propose several enhancements to a geometric transformation based model for anomaly detection in images (GeoTranform). The model assumes that the anomaly class is unknown and that only inlier samples are available for training. We introduce new filter based transformations useful for detecting anomalies in astronomical images, that highlight artifact properties to make them more easily distinguishable from real objects. In addition, we propose a transformation selection strategy that allows us to find indistinguishable pairs of transformations. This results in an improvement of…
Intelligent scheduling of the sequence of scientific exposures taken at ground-based astronomical observatories is massively challenging. Observing time is over-subscribed and atmospheric conditions are constantly changing. We propose to guide observatory scheduling using machine learning. Leveraging a 15-year archive of exposures, environmental, and operating conditions logged by the Canada-France-Hawaii Telescope, we construct a probabilistic data-driven model that accurately predicts image quality. We demonstrate that, by optimizing the opening and closing of twelve vents placed on the dome…
To reduce and analyze astronomical images, astronomers can rely on a wide range of libraries providing low-level implementations of legacy algorithms. However, combining these routines into robust and functional pipelines requires a major effort which often ends up in instrument-specific and poorly maintainable tools, yielding products that suffer from a low-level of reproducibility and portability. In this context, we present prose, a Python framework to build modular and maintainable image processing pipelines. Built for astronomy, it is instrument-agnostic and allows the construction of pip…
Written in Python and utilising ParselTongue to interface with the Astronomical Image Processing System (AIPS), the e-MERLIN data reduction pipeline is intended to automate the procedures required in processing and calibrating radio astronomy data from the e-MERLIN correlator. Driven by a plain text file of input parameters, the pipeline is modular and can be run in stages by the user, depending on requirements. The software includes options to load raw data, average in time and/or frequency, flag known sources of interference, flag more comprehensively with SERPent, carry out some or all of t…
We propose a new sequential classification model for astronomical objects based on a recurrent convolutional neural network (RCNN) which uses sequences of images as inputs. This approach avoids the computation of light curves or difference images. This is the first time that sequences of images are used directly for the classification of variable objects in astronomy. The second contribution of this work is the image simulation process. We generate synthetic image sequences that take into account the instrumental and observing conditions, obtaining a realistic, set of movies for each astronomi…
We have discovered an anomalous behavior of CCD readout electronics that affects their use in many astronomical applications. An offset in the digitization of the CCD output voltage that depends on the binary encoding of one pixel is added to pixels that are read out one, two and/or three pixels later. One result of this effect is the introduction of a differential offset in the background when comparing regions with and without flux from science targets. Conventional data reduction methods do not correct for this offset. We find this effect in 16 of 22 instruments investigated, covering a var…
ImageJ is a graphical user interface (GUI) driven, public domain, Java-based, software package for general image processing traditionally used mainly in life sciences fields. The image processing capabilities of ImageJ are useful and extendable to other scientific fields. Here we present AstroImageJ (AIJ), which provides an astronomy specific image display environment and tools for astronomy specific image calibration and data reduction. Although AIJ maintains the general purpose image processing capabilities of ImageJ, AIJ is streamlined for time-series differential photometry, light curve de…
ImageJ is a graphical user interface (GUI) driven, public domain, Java-based, software package for general image processing traditionally used mainly in life sciences fields. The image processing capabilities of ImageJ are useful and extendable to other scientific fields. Here we present AstroImageJ (AIJ), which provides an astronomy specific image display environment and tools for astronomy specific image calibration and data reduction. Although AIJ maintains the general purpose image processing capabilities of ImageJ, AIJ is streamlined for time-series differential photometry, light curve de…
The sensitivity of the gravitational-wave detector KAGRA, presently under construction, will be limited by quantum noise in a large fraction of its spectrum. The most promising technique to increase the detector sensitivity is the injection of squeezed states of light, where the squeezing angle is dynamically rotated by a Fabry-Pérot filter cavity. One of the main issues in the filter cavity design and realization is the optical losses due to the mirror surface imperfections. In this work we present a study of the specifications for the mirrors to be used in a 300 m filter cavity for the KAGRA…
The Image Reduction and Analysis Facility is a scientific image processing package widely used throughout the astronomical community. IRAF has been developed and distributed by the National Optical Astronomy Observatory in Tucson, Arizona since the early 1980's. Other observatories and projects have written many dozens of layered external application packages. More than ten thousand journal articles acknowledge the use of IRAF and thousands of professional astronomers rely on it. As with many other classes of astronomical software, IRAF depends on Universal Time (UT) in many modules throughout…
Istituto di Radio Astronomia, INAF（意大利国家天体物理研究所射电天文研究所）；Hamburger Sternwarte（汉堡天文台）；Universitá di Bologna（博洛尼亚大学）
We apply a Machine Learning technique known as Convolutional Denoising Autoencoder to denoise synthetic images of state-of-the-art radio telescopes, with the goal of detecting the faint, diffused radio sources predicted to characterise the radio cosmic web. In our application, denoising is intended to address both the reduction of random instrumental noise and the minimisation of additional spurious artefacts like the sidelobes, resulting from the aperture synthesis technique. The effectiveness and the accuracy of the method are analysed for different kinds of corrupted input images, together …
Current optical space telescopes rely upon silicon Charge Coupled Devices (CCDs) to detect and image the incoming photons. The performance of a CCD detector depends on its ability to transfer electrons through the silicon efficiently, so that the signal from every pixel may be read out through a single amplifier. This process of electron transfer is highly susceptible to the effects of solar proton damage (or non-ionizing radiation damage). This is because charged particles passing through the CCD displace silicon atoms, introducing energy levels into the semi-conductor bandgap which act as lo…
We present a new procedure rooted in deep learning to construct science images from data cubes collected by astronomical instruments using HxRG detectors in low-flux regimes. It improves on the drawbacks of the conventional algorithms to construct 2D images from multiple readouts by using the readout scheme of the detectors to reduce the impact of correlated readout noise. We train a convolutional recurrent neural network on simulated astrophysical scenes added to laboratory darks to estimate the flux on each pixel of science images. This method achieves a reduction of the noise on constructed…
Trend filtering---first introduced into the astronomical literature in Paper I of this series---is a state-of-the-art statistical tool for denoising one-dimensional signals that possess varying degrees of smoothness. In this work, we demonstrate the broad utility of trend filtering to observational astronomy by discussing how it can contribute to a variety of spectroscopic and time-domain studies. The observations we discuss are (1) the Lyman-$α$ forest of quasar spectra; (2) more general spectroscopy of quasars, galaxies, and stars; (3) stellar light curves with planetary transits; (4) eclips…
We have been developing monolithic active pixel sensors, known as Kyoto's X-ray SOIPIXs, based on the CMOS SOI (silicon-on-insulator) technology for next-generation X-ray astronomy satellites. The event trigger output function implemented in each pixel offers microsecond time resolution and enables reduction of the non-X-ray background that dominates the high X-ray energy band above 5--10 keV. A fully depleted SOI with a thick depletion layer and back illumination offers wide band coverage of 0.3--40 keV. Here, we report recent progress in the X-ray SOIPIX development. In this study, we achiev…
Modern interferometers routinely provide radio-astronomical images down to subarcsecond resolution. However, interferometers filter out spatial scales larger than those sampled by the shortest baselines, which affects the measurement of both spatial and spectral features. Complementary single-dish data are vital for recovering the true flux distribution of spatially resolved astronomical sources with such extended emission. In this work, we provide an overview of the prominent available methods to combine single-dish and interferometric observations. We test each of these methods in the framew…
We present an application of autoencoders to the problem of noise reduction in single-shot astronomical images and explore its suitability for upcoming large-scale surveys. Autoencoders are a machine learning model that summarises an input to identify its key features, then from this knowledge predicts a representation of a different input. The broad aim of our autoencoder model is to retain morphological information (e.g., non-parametric morphological information) from the survey data whilst simultaneously reducing the noise contained in the image. We implement an autoencoder with convolution…
Earth-based gravitational-wave detectors will be limited by quantum noise in a large part of their spectrum. The most promising technique to achieve a broadband reduction of such noise is the injection of a frequency dependent squeezed vacuum state from the output port of the detector, whit the squeeze angle rotated by the reflection off a Fabry-Perot filter cavity. One of the most important parameters limiting the squeezing performance is represented by the optical losses of the filter cavity. We report here the operation of a 300 m filter cavity prototype installed at the National Astronomic…
Astronomical CCD systems based on second-generation DINACON controllers were developed at the SAO RAS Advanced Design Laboratory more than seven years ago and since then have been in constant operation at the 6-meter and Zeiss-1000 telescopes. Such systems use monolithic large-area CCDs. We describe the software developed for the control of a family of large-area CCD systems equipped with a DINACON-II controller. The software suite serves for acquisition, primary reduction, visualization, and storage of video data, and also for the control, setup, and diagnostics of the CCD system.
We present the first public version of SImMER, an open-source Python reduction pipeline for astronomical images of point sources. Current capabilities include dark-subtraction, flat-fielding, sky-subtraction, image registration, FWHM measurement, contrast curve calculation, and table and plot generation. SImMER supports observations taken with the ShARCS camera on the Shane 3-m telescope and the PHARO camera on the Hale 5.1-m telescope. The modular nature of SImMER allows users to extend the pipeline to accommodate additional instruments with relative ease. One of the core functions of the pip…
PhoPS: An automated photometric pipeline for survey-era astronomy
Türkiye National Observatories, TUG（土耳其国家天文台）；The Scientific and Technological Research Council of Türkiye (TÜBİTAK)（土耳其科学技术研究委员会）；Instituto de Astrofísica de Andalucía (IAA-CSIC)（安达卢西亚天体物理研究所）；LIRA, CNRS UMR8254, Observatoire de Paris（巴黎天文台）
