With ever increasing data rates produced by modern radio telescopes like LOFAR and future telescopes like the SKA, many data processing steps are overwhelmed by the amount of data that needs to be handled using limited compute resources. Calibration is one such operation that dominates the overall data processing computational cost, nonetheless, it is an essential operation to reach many science goals. Calibration algorithms do exist that scale well with the number of stations of an array and the number of directions being calibrated. However, the remaining bottleneck is the raw data volume, which scales with the number of baselines, and which is proportional to the square of the number of stations. We propose a 'stochastic' calibration strategy where we only read in a mini-batch of data for obtaining calibration solutions, as opposed to reading the full batch of data being calibrated. Nonetheless, we obtain solutions that are valid for the full batch of data. Normally, data need to be averaged before calibration is performed to accommodate the data in size-limited compute memory. Stochastic calibration overcomes the need for data averaging before any calibration can be performed, and offers many advantages including: enabling the mitigation of faint radio frequency interference; better removal of strong celestial sources from the data; and better detection and spatial localization of fast radio transients.
The calibration of modern radio interferometers is a significant challenge, specifically at low frequencies. In this perspective, we propose a novel iterative calibration algorithm, which employs the popular sparse representation framework, in the regime where the propagation conditions shift dissimilarly the directions of the sources. More precisely, our algorithm is designed to estimate the apparent directions of the calibration sources, their powers, the directional and undirectional complex gains of the array elements and their noise powers, with a reasonable computational complexity. Nume…
In order to meet the theoretically achievable imaging performance, calibration of modern radio interferometers is a mandatory challenge, especially at low frequencies. In this perspective, we propose a novel parallel iterative multi-wavelength calibration algorithm. The proposed algorithm estimates the apparent directions of the calibration sources, the directional and undirectional complex gains of the array elements and their noise powers, with a reasonable computational complexity. Furthermore, the algorithm takes into account the specific variation of the aforementioned parameter values ac…
In radio astronomy, accurate calibration is of crucial importance for the new generation of radio interferometers. More specifically, because of the potential presence of outliers which affect the measured data, robustness needs to be ensured. On the other hand, calibration is improved by taking advantage of these new instruments and exploiting the known structure of parameters of interest across frequency. Therefore, we propose in this paper an iterative robust multi-frequency calibration algorithm based on a distributed and consensus optimization scheme which aims to estimate the complex gai…
High precison calibration is essential for a new generation of radio interferometers looking for Epoch of Reionization and Baryon Acoustic Oscillation signatures in neutral hydrogen. These arrays have so far been calibrated by redundant calibration, which usually assumes baselines intended to be identical are perfectly so. We present a new calibration scheme that relaxes the assumption of explicit redundancy by calculating the expected covariance of baselines. The technique also allows one to take advantage of partial knowledge of the sky, such as point sources with known positions but unknown…
Growing interest in 21 cm tomography has led to the design and construction of broadband radio interferometers with low noise, moderate angular resolution, high spectral resolution, and wide fields of view. With characteristics somewhat different from traditional radio instruments, these interferometers may require new calibration techniques in order to reach their design sensitivities. Self-calibration or redundant calibration techniques that allow an instrument to be calibrated off complicated sky emission structures are ideal. In particular, the large number of redundant baselines possessed…
In this paper, we investigate the calibration of radio interferometers in which Jones matrices are considered to model the interaction between the incident electromagnetic field and the antennas of each station. Specifically, perturbation effects are introduced along the signal path, leading to the conversion of the plane wave into an electric voltage by the receptor. In order to design a robust estimator, the noise is assumed to follow a spherically invariant random process (SIRP). The derived algorithm is based on an iterative relaxed concentrated maximum likelihood estimator (MLE), for whic…
Having an accurate calibration method is crucial for any scientific research done by a radio telescope. The next generation radio telescopes such as the Square Kilometre Array (SKA) will have a large number of receivers which will produce exabytes of data per day. In this paper we propose new direction-dependent and independent calibration algorithms that, while requiring much less storage during calibration, converge very fast. The calibration problem can be formulated as a non-linear least square optimization problem. We show that combining a block-LDU decomposition with Gauss-Newton iterati…
Radio interferometers can measure the full polarization state of incoming waves by cross--correlating all combinations of two orthogonal polarizations at each antenna. The independent sets of electronics used to detect the two polarization states will introduce a differential instrumental delay between the two data streams. The usual technique of separate calibration of the parallel--hand sets of visibilities still allows for an arbitrary offset in group delay and phase between the two parallel systems. In order to use the cross--polarized visibilities, this instrumental offset must be determi…
We study the impact of sky-based calibration errors from source mismodeling on 21\,cm power spectrum measurements with an interferometer and propose a method for suppressing their effects. While emission from faint sources that are not accounted for in calibration catalogs is believed to be spectrally smooth, deviations of true visibilities from model visibilities are not, due to the inherent chromaticity of the interferometer's sky-response (the "wedge"). Thus, unmodeled foregrounds, below the confusion limit of many instruments, introduce frequency structure into gain solutions on the same l…
We use information theory to derive fundamental limits on the capacity to calibrate next-generation radio interferometers, and measure parameters of point sources for instrument calibration, point source subtraction, and data deconvolution. We demonstrate the implications of these fundamental limits, with particular reference to estimation of the 21cm Epoch of Reionization power spectrum with next-generation low-frequency instruments (e.g., the Murchison Widefield Array -- MWA, Precision Array for Probing the Epoch of Reionization -- PAPER), where short time scale instrumental calibration is r…
In this paper, we aim to design robust estimation techniques based on the compound-Gaussian (CG) process and adapted for calibration of radio interferometers. The motivation beyond this is due to the presence of outliers leading to an unrealistic traditional Gaussian noise assumption. Consequently, to achieve robustness, we adopt a maximum a posteriori (MAP) approach which exploits Bayesian statistics and follows a sequential updating procedure here. The proposed algorithm is applied in a multi-frequency scenario in order to enhance the estimation and correction of perturbation effects. Numeri…
This paper investigates calibration of sensor arrays in the radio astronomy context. Current and future radio telescopes require computationally efficient algorithms to overcome the new technical challenges as large collecting area, wide field of view and huge data volume. Specifically, we study the calibration of radio interferometry stations with significant direction dependent distortions. We propose an iterative robust calibration algorithm based on a relaxed maximum likelihood estimator for a specific context: i) observations are affected by the presence of outliers and ii) parameters of …
We propose a new method to measure gravitational redshift effect using simultaneous interferometric observations of a distant radio source to synchronize clocks. The first order by $v/c$ contribution to the signal (the classical Doppler effect) is automatically canceled in our setup. When other contributions from the velocities of the clocks, clock imperfection and atmosphere are properly taken into account, the residual gravitational redshift can be measured with the relative precision of $\sim 10^{-3}$ for RadioAstron space-to-ground interferometer or with precision up to few $10^{-5}$ with …
Radio interferometer arrays with non-homogeneous element patterns are more difficult to calibrate compared to the more common homogeneous array. In particular, the non-homogeneity of the patterns has significant implications on the computational tractability of evaluating the calibration solutions. We apply the A-stacking technique to this problem and explore the trade-off to be made between the calibration accuracy and computational complexity. Through simulations, we show that this technique can be favourably applied in the context of an SKA-Low station. We show that the minimum accuracy req…
Paper I of the series re-derived the radio interferometry measurement equation (RIME) from first principles, and extended the Jones formalism to the full-sky case, incorporating direction-dependent effects (DDEs). This paper aims to describe both classical radio interferometric calibration (selfcal and related methods), and the recent developments in the treatment of DDEs, using the RIME-based mathematical framework developed in Paper I. It also aims to demonstrate the ease with which the various effects can be described and understood. The first section of this paper uses the RIME formalism t…
Future generations of radio interferometers targeting the 21\,cm signal at cosmological distances with $N\gg 1000$ antennas could face a significant computational challenge in building correlators with the traditional architecture, whose computational resource requirement scales as $\mathcal{O}(N^2)$ with array size. The fundamental output of such correlators is the cross-correlation products of all antenna pairs in the array. The FFT-correlator architecture reduces the computational resources scaling to $\mathcal{O}(N\log{N})$ by computing cross-correlation products through a spatial Fourier …
Global 21-cm experiments require exquisitely precise calibration of the measurement systems in order to separate the weak 21-cm signal from Galactic and extragalactic foregrounds as well as instrumental systematics. Hitherto, experiments aiming to make this measurement have concentrated on measuring this signal using the single element approach. However, an alternative approach based on interferometers with short baselines is expected to alleviate some of the difficulties associated with a single element approach such as precision modelling of the receiver noise spectrum. Short spacing Interfe…
This manual is intended to help ALMA and other interferometer users improve images by recognising limitations and how to overcome them and deciding when and how to use self-calibration. The images provided by the ALMA Science Archive are calibrated using standard observing and data processing routines, including a quality assurance process to make sure that the observations meet the proposer's science requirements. This may not represent the full potential of the data, since any interferometry observation can be imaged with a range of resolutions and surface brightness sensitivity. The separat…
The response of the Advanced LIGO interferometers is known to vary with time [arXiv:1608.05134]. Accurate calibration of the interferometers must therefore track and compensate for temporal variations in calibration model parameters. These variations were tracked during the first three Advanced LIGO observing runs, and compensation for some of them has been implemented in the calibration procedure. During the second observing run, multiplicative corrections to the interferometer response were applied while producing calibrated strain data both in real-time and in high-latency. In a high-latenc…
Redundant calibration is a technique in radio astronomy that allows calibration of radio arrays whose antennas lie on a lattice by exploiting the fact that redundant baselines should see the same sky signal. Because the number of measured visibilities scales quadratically with the number of antennas but the number of unknowns describing the individual antenna responses and the available information about the sky scales only linearly with the array size, the problem is always over-constrained as long as the array is big and dense enough. This is true even for non-lattice array configurations. I…
Interferometry with phased-array feeds (PAFs) presents new calibration challenges in comparison with single-pixel feeds. In particular, temporal instability of the compound beam patterns due to element gain drifts (EGDs) can produce calibration artefacts in interferometric images. To translate imaging dynamic range requirements into PAF hardware and calibration requirements, we must learn to relate EGD levels to imaging artefact levels. We present a MeqTrees-based simulations framework that addresses this problem, and apply it to the APERTIF prototype currently in development for the WSRT.
The requirements on the stability of the frequency reference in the Square Kilometre Array (SKA), as a radio astronomy interferometer, are given in terms of maximum accepted degree of coherence loss caused by the instability of the frequency reference. In this paper we analyse the relationship between the characterisation of the instability of the frequency reference in the radio astronomy array and the coherence loss. The calculation of the coherence loss from the instability characterisation given by the Allan deviation is reviewed. Some practical aspects and limitations are analysed.
We present a filtering technique that can be applied to individual baselines of wide-bandwidth, wide-field interferometric data to geometrically select regions on the celestial sphere that contain primary calibration sources. The technique relies on the Fourier transformation of wide-band frequency spectra from a given baseline to obtain one-dimensional "delay images", and then the transformation of a time-series of delay images to obtain two-dimensional "delay/delay-rate images." Source selection is possible in these images given appropriate combinations of baseline, bandwidth, integration ti…
RadioAstron is a Russian space based radio telescope with a ten meter dish in a highly elliptical orbit with an eight to nine day period. RadioAstron works together with Earth based radio telescopes to give interferometer baselines extending up to 350,000 km, more than an order of magnitude improvement over what is possible from earth based very long baseline interferometry. Operating in four frequency bands, 1.3, 6, 18, and 92 cm, the corresponding resolutions are 7, 35, 100, and 500 microarcseconds respectively in the four wavelength bands.
We report development of a simple and affordable radio interferometer suitable as an educational laboratory experiment. With the increasing importance of interferometry in astronomy, the lack of educational interferometers is an obstacle to training the future generation of astronomers. This interferometer provides the hands-on experience needed to fully understand the basic concepts of interferometry. The design of this interferometer is based on the Michelson & Pease stellar optical interferometer, but operates at a radio wavelength (~11 GHz; ~2.7cm); thus the requirement for optical accurac…
From the Virgo interferometer calibration to the bias and uncertainty of the h(t) detector strain during the O4 run
Université de Strasbourg, CNRS, IPHC UMR 7178（斯特拉斯堡大学）；Laboratoire d’Annecy de Physique des Particules (LAPP), Univ. Grenoble Alpes, Université Savoie Mont Blanc, CNRS/IN2P3（阿讷西粒子物理实验室）；Institut de Fisica d’Altes Energies (IFAE), Barcelona Institute of Science and Technology, and ICREA（高能物理研究所）；Universiteit Antwerpen（安特卫普大学）
Gravitational-wave (GW) laser interferometers such as Advanced LIGO transduce spacetime strain into optical power fluctuation. Converting this optical power fluctuations back into an estimated spacetime strain requires a calibration process that accounts for both the interferometer's optomechanical response and the feedback control loop used to control the interferometer test masses. Systematic errors in the calibration parameters lead to systematic errors in the GW strain estimate, and hence to systematic errors in the astrophysical parameter estimates in a particular GW signal. In this work …
We describe techniques concerning wavelength calibration and sky subtraction to maximise the scientific utility of data from tunable filter instruments. While we specifically address data from the Optical System for Imaging and low Resolution Integrated Spectroscopy instrument (OSIRIS) on the 10.4~m Gran Telescopio Canarias telescope, our discussion is generalisable to data from other tunable filter instruments. A key aspect of our methodology is a coordinate transformation to polar coordinates, which simplifies matters when the tunable filter data is circularly symmetric around the optical ce…
Increasing data volumes delivered by a new generation of radio interferometers require computationally efficient and robust calibration algorithms. In this paper, we propose distributed calibration as a way of improving both computational cost as well as robustness in calibration. We exploit the data parallelism across frequency that is inherent in radio astronomical observations that are recorded as multiple channels at different frequencies. Moreover, we also exploit the smoothness of the variation of calibration parameters across frequency. Data parallelism enables us to distribute the comp…
Aims: This paper aims to derive an expression for the sensitivity of a polarimetric radio interferometer that is valid for all-sky observations of arbitrarily polarized sources, with neither a restriction on FoV nor with any a priori assumption regarding the polarization state of the source. We verify the resulting formula with an all-sky observation using the Murchison Widefield Array (MWA) telescope. Methods: The sensitivity expression is developed from first principles by applying the concept of System Equivalent Flux Density (SEFD) to a polarimetric radio interferometer not by computing $A…
Distributed calibration based on consensus optimization is a computationally efficient method to calibrate large radio interferometers such as LOFAR and SKA. Calibrating along multiple directions in the sky and removing the bright foreground signal is a crucial step in many science cases in radio interferometry. The residual data contain weak signals of huge scientific interest and of particular concern is the effect of incomplete sky models used in calibration on the residual. In order to study this, we consider the mapping between the input uncalibrated data and the output residual data. We …
The formulation of the radio interferometer measurement equation (RIME) by Hamaker et al. has provided us with an elegant mathematical apparatus for better understanding, simulation and calibration of existing and future instruments. The calibration of the new radio telescopes (LOFAR, SKA) would be unthinkable without the RIME formalism, and new software to exploit it. MeqTrees is designed to implement numerical models such as the RIME, and to solve for arbitrary subsets of their parameters. The technical goal of MeqTrees is to provide a tool for rapid implementation of such models, while offe…
We propose a new technique to obtain super-resolution images with radio interferometer using sparse modeling. In standard radio interferometry, sampling of ($u$, $v$) is quite often incomplete and thus obtaining an image from observed visibilities becomes an underdetermined problem, and a technique so-called "zero-padding" is often used to fill up unsampled grids in ($u$, $v$) plane, resulting in image degradation by finite beam size as well as numerous side-lobes. In this paper we show that directly solving such an underdetermined problem based on sparse modeling (in this paper LASSO) avoids …
Radio interferometers consisting of identical antennas arranged on a regular lattice permit fast Fourier transform beamforming, which reduces the correlation cost from $\mathcal{O}(n^2)$ in the number of antennas to $\mathcal{O}(n\log n)$. We develop a formalism for describing this process and apply this formalism to derive a number of algorithms with a range of observational applications. These include algorithms for forming arbitrarily pointed tied-array beams from the regularly spaced Fourier-transform formed beams, sculpting the beams to suppress sidelobes while only losing percent-level s…
Distributed radio interferometric calibration based on consensus optimization has been shown to improve the estimation of systematic errors in radio astronomical observations. The intrinsic continuity of systematic errors across frequency is used by a consensus polynomial to penalize traditional calibration. Consensus is achieved via the use of alternating direction method of multipliers (ADMM) algorithm. In this paper, we extend the existing distributed calibration algorithms to use ADMM with an adaptive penalty parameter update. Compared to a fixed penalty, its adaptive update has been shown…
The Radio Interferometer Measurement Equation (RIME) is a matrix-based mathematical model that describes the response of a radio interferometer. The Jones calculus it employs is not suitable for describing the analogue components of a telescope. This is because it does not consider the effect of impedance mismatches between components. This paper aims to highlight the limitations of Jones calculus, and suggests some alternative methods that are more applicable. We reformulate the RIME with a different basis that includes magnetic and mixed coherency statistics. We present a microwave network i…
Calibration of a typical radio interferometric array yields thousands of parameters as solutions. These solutions contain valuable information about the systematic errors in the data (ionosphere and beam shape). This information could be reused in calibration to improve the accuracy and also can be fed into imaging to improve the fidelity. We propose a distributed optimization strategy to construct models for the systematic errors in the data using the calibration solutions. We formulate this as an elastic net regularized distributed optimization problem which we solve using the alternating di…
The 21-cm signal of neutral hydrogen - emitted during the Epoch of Reionization - promises to be an important source of information for the study of the infant universe. However, its detection is impossible without sufficient mitigation of other strong signals in the data, which requires an accurate knowledge of the instrument. Using the result of instrument calibration, a large part of the contaminating signals are removed and the resulting residual data is further analyzed in order to detect the 21-cm signal. Direction dependent calibration (DDC) can strongly affect the 21-cm signal, however…
We discuss the ongoing development of single-mode fiber Fabry-Perot (FFP) Interferometers as precise astro-photonic calibration sources for high precision radial velocity (RV) spectrographs. FFPs are simple, inexpensive, monolithic units that can yield a stable and repeatable output spectrum. An FFP is a unique alternative to a traditional etalon, as the interferometric cavity is made of single-mode fiber rather than an air-gap spacer. This design allows for excellent collimation, high spectral finesse, rigid mechanical stability, insensitivity to vibrations, and no need for vacuum operation. …
We recently proposed the use of consensus optimization as a viable and effective way to improve the quality of calibration of radio interferometric data. We showed that it is possible to obtain far more accurate calibration solutions and also to distribute the compute load across a network of computers by using this technique. A crucial aspect in any consensus optimization problem is the selection of the penalty parameter used in the alternating direction method of multipliers (ADMM) iterations. This affects the convergence speed as well as the accuracy. In this paper, we use the Hessian of th…
Papers I and II of this series have extended the radio interferometry measurement equation (RIME) formalism to the full-sky case, and provided a RIME-based description of calibration and the problem of direction-dependent effects (DDEs). This paper aims to provide a practical demonstration of a RIME-based approach to calibration, via an example of extremely high-dynamic range calibration of WSRT observations of 3C 147 at 21 cm, with full treatment of DDEs. A version of the RIME incorporating differential gains has been implemented in MeqTrees, and applied to the 3C 147 data. This was used to p…
Spectral features introduced by instrumental chromaticity of radio interferometers have the potential to negatively impact the ability to perform Epoch of Reionisation (EoR) and Cosmic Dawn (CD) science using the redshifted neutral hydrogen emission line from the early Universe. We describe instrument calibration choices that influence the spectral characteristics of the science data, and assess their impact on EoR statistical and tomographic experiments. Principally, we consider the intrinsic spectral response of the receiving antennas, embedded within a complete frequency-dependent primary b…
For a ground-based radio interferometer observing at low frequencies, the ionosphere causes propagation delays and refraction of cosmic radio waves which result in phase errors in the received signal. These phase errors can be corrected using a calibration method that assumes a two-dimensional phase screen at a fixed altitude above the surface of the Earth, known as the thin-layer model. Here we investigate the validity of the thin-layer model and provide a simple equation with which users can check when this approximation can be applied to observations for varying time of day, zenith angle, i…
Ground based low frequency radio interferometers have been developed in the last decade and are providing the scientific community with high quality observations. Conversely, current radioastronomy instruments in space have a poor angular resolution with single point observation systems. Improving the observation capabilities of the low frequency range (a few kHz to 100 MHz) requires to go to space and to set up a space based network of antenna that can be used as an interferometer. We present the outcome of the NOIRE (Nanosatellites pour un Observatoire Interférométrique Radio dans l'Espace /…
The knowledge of receiver beam shapes is essential for accurate radio interferometric imaging. Traditionally, this information is obtained by holographic techniques or by numerical simulation. However, such methods are not feasible for an observation with time varying beams, such as the beams produced by a phased array radio interferometer. We propose the use of the observed data itself for the estimation of the beam shapes. We use the directional gains obtained along multiple sources across the sky for the construction of a time varying beam model. The construction of this model is an ill pos…
Upgrading the infrastructure of old scientific instruments requires the development of new hardware and software which may be expensive (in general, these projects lack of enough resources to acquire fast and modern infrastructure to become ompetitive and functional in the era of digital data). Particularly, the development of software for data acquisition in real time is one of the most important topics in experimental science and industry. However, the constant improvements on the data acquisition hardware, discourages the development of software highly optimized in order to minimize the con…
Radio halos are diffuse, extended sources of radio emission detected primarily in massive, merging galaxy clusters. In smaller and/or relaxed clusters, where no halos are detected, one can instead place upper limits to a possible radio emission. Detections and upper limits are both crucial to constrain theoretical models for the generation of radio halos. The upper limits are model dependent for radio interferometers and thus the process of obtaining these is tedious to perform manually. In this paper, we present a Python based tool to automate this process of estimating the upper limits. The …
University of Cape Town（开普敦大学）；University of the Western Cape（西开普大学）；Rhodes University（罗德斯大学）；Observatoire de Paris（巴黎天文台）；CNRS（法国国家科学研究中心）；Université Paris Diderot（巴黎狄德罗大学）；SKA South Africa（SKA南非）
We present Montblanc, a GPU implementation of the Radio interferometer measurement equation (RIME) in support of the Bayesian inference for radio observations (BIRO) technique. BIRO uses Bayesian inference to select sky models that best match the visibilities observed by a radio interferometer. To accomplish this, BIRO evaluates the RIME multiple times, varying sky model parameters to produce multiple model visibilities. Chi-squared values computed from the model and observed visibilities are used as likelihood values to drive the Bayesian sampling process and select the best sky model. As mos…
We report on the MIT Epoch of Reionization (MITEoR) experiment, a pathfinder low-frequency radio interferometer whose goal is to test technologies that improve the calibration precision and reduce the cost of the high-sensitivity 3D mapping required for 21 cm cosmology. MITEoR accomplishes this by using massive baseline redundancy, which enables both automated precision calibration and correlator cost reduction. We demonstrate and quantify the power and robustness of redundancy for scalability and precision. We find that the calibration parameters precisely describe the effect of the instrumen…
Context. Frequent, simultaneous observations across the electromagnetic spectrum are essential to the study of a range of astrophysical phenomena including Active Galactic Nuclei. A key tool of such studies is the ability to observe an object when it flares i.e. exhibits a rapid and significant increase in its flux density. Aims. We describe the specific observational procedures and the calibration techniques that have been developed and tested to create a single baseline radio interferometer that can rapidly observe a flaring object. This is the only facility that is dedicated to rapid high r…
We introduce CALAMITY, a precision bandpass calibration method for radio interferometry. CALAMITY can solve for direction independent gains with arbitrary frequency structure to the high precision required for 21 cm cosmology with minimal knowledge of foregrounds or antenna beams and does not require any degree of redundancy (repeated identical measurements of the same baseline). We have achieved this through two key innovations. Firstly, we model the foregrounds on each baseline independently using a flexible and highly efficient set of basis functions that have minimal overlap with 21 cm mod…
In order to cope with the increased data volumes generated by modern radio interferometers such as LOFAR (Low Frequency Array) or SKA (Square Kilometre Array), fast and efficient calibration algorithms are essential. Traditional radio interferometric calibration is performed using nonlinear optimization techniques such as the Levenberg-Marquardt algorithm in Euclidean space. In this paper, we reformulate radio interferometric calibration as a nonlinear optimization problem on a Riemannian manifold. The reformulated calibration problem is solved using the Riemannian trust-region method. We show…
