Radio astronomy is entering a new era with new and future radio observatories such as the Low Frequency Array and the Square Kilometer Array. We describe in detail an automated flagging pipeline and evaluate its performance. With only a fraction of the computational cost of correlation and its use of the previously introduced SumThreshold method, it is found to be both fast and unrivalled in its high accuracy. The LOFAR radio environment is analysed with the help of this pipeline. The high time and spectral resolution of LOFAR have resulted in an observatory where only a few percent of the data is lost due to RFI.
Radio Frequency Interference (RFI) corrupts astronomical measurements, thus affecting the performance of radio telescopes. To address this problem, supervised segmentation models have been proposed as candidate solutions to RFI detection. However, the unavailability of large labelled datasets, due to the prohibitive cost of annotating, makes these solutions unusable. To solve these shortcomings, we focus on the inverse problem; training models on only uncontaminated emissions thereby learning to discriminate RFI from all known astronomical signals and system noise. We use Nearest-Latent-Neighb…
The rapid development of new generation radio interferometers such as the Square Kilometer Array (SKA) has opened up unprecedented opportunities for astronomical research. However, anthropogenic Radio Frequency Interference (RFI) from communication technologies and other human activities severely affects the fidelity of observational data. It also significantly reduces the sensitivity of the telescopes. We proposed a robust Convolutional Neural Network (CNN) model to identify RFI based on machine learning methods. We overlaid RFI on the simulation data of SKA1-LOW to construct three visibility…
We describe the design and implementation of an extremely scalable real-time RFI mitigation method, based on the offline AOFlagger. All algorithms scale linearly in the number of samples. We describe how we implemented the flagger in the LOFAR real-time pipeline, on both CPUs and GPUs. Additionally, we introduce a novel simple history-based flagger that helps reduce the impact of our small window on the data. By examining an observation of a known pulsar, we demonstrate that our flagger can achieve much higher quality than a simple thresholder, even when running in real time, on a distributed …
LOFAR is a leading aperture synthesis telescope operated in the Netherlands with stations across Europe. The LOFAR Two-meter Sky Survey (LoTSS) will produce more than 3000 14 TB data sets, mapping the entire northern sky at low frequencies. The data produced by this survey is important for understanding the formation and evolution of galaxies, supermassive black holes and other astronomical phenomena. All of the LoTSS data needs to be processed by the LOFAR Direction Independent (DI) pipeline, prefactor. Understanding the performance of this pipeline is important when trying to optimize the th…
Modern astronomical data processing requires complex software pipelines to process ever growing datasets. For radio astronomy, these pipelines have become so large that they need to be distributed across a computational cluster. This makes it difficult to monitor the performance of each pipeline step. To gain insight into the performance of each step, a performance monitoring utility needs to be integrated with the pipeline execution. In this work we have developed such a utility and integrated it with the calibration pipeline of the Low Frequency Array, LOFAR, a leading radio telescope. We te…
LOFAR is a new and sensitive radio interferometer that can be used for dynamic high-resolution imaging spectroscopy at low radio frequencies from 10 to 90 and 110 to 250 MHz. Here we describe its usage for observations of the Sun and in particular of solar radio bursts. We also describe the processing, archiving and accessing of solar LOFAR data, which is accomplished via the LOFAR Solar Imaging Pipeline and the LOFAR Solar Data Center.
Searches for millisecond-duration, dispersed single pulses have become a standard tool used during radio pulsar surveys in the last decade. They have enabled the discovery of two new classes of sources: rotating radio transients and fast radio bursts. However, we are now in a regime where the sensitivity to single pulses in radio surveys is often limited more by the strong background of radio frequency interference (RFI, which can greatly increase the false-positive rate) than by the sensitivity of the telescope itself. To mitigate this problem, we introduce the Single-pulse Searcher (SpS). Th…
The new generation of high-resolution broad-band radio telescopes, like the Low Frequency Array (LOFAR), produces, depending on the level of compression, between 1 to 10 TB of data per hour after correlation. Such a large amount of scientific data demand powerful computing resources and efficient data handling strategies to be mastered. The LOFAR Two-metre Sky Survey (LoTSS) is a Key Science Project (KSP) of the LOFAR telescope. It aims to map the entire northern hemisphere at unprecedented sensitivity and resolution. The survey consist of 3 168 pointings, requiring about 30 PBytes of storage …
In this paper, we present a framework for assessing the effect of non-stationary Gaussian noise and radio frequency interference (RFI) on the signal to noise ratio, the number of false positives detected per true positive and the sensitivity of standard pulsar search pipelines. The results highlight the necessity to develop algorithms that are able to identify and remove non-stationary variations from the data before RFI excision and searching is performed in order to limit false positive detections. The results also show that the spectrum whitening algorithms currently employed, severely affe…
The low flux of the ultra-high energy cosmic rays (UHECR) at the highest energies provides a challenge to answer the long standing question about their origin and nature. Even lower fluxes of neutrinos with energies above $10^{22}$ eV are predicted in certain Grand-Unifying-Theories (GUTs) and e.g.\ models for super-heavy dark matter (SHDM). The significant increase in detector volume required to detect these particles can be achieved by searching for the nano-second radio pulses that are emitted when a particle interacts in Earth's moon with current and future radio telescopes. In this contri…
With the arrival of a number of wide-field snapshot image-plane radio transient surveys, there will be a huge influx of images in the coming years making it impossible to manually analyse the datasets. Automated pipelines to process the information stored in the images are being developed, such as the LOFAR Transients Pipeline, outputting light curves and various transient parameters. These pipelines have a number of tuneable parameters that require training to meet the survey requirements. This paper utilises both observed and simulated datasets to demonstrate different machine learning strat…
We present VOLKS2, the second release of "VLBI Observation for transient Localization Keen Searcher". The pipeline aims at transient search in regular VLBI observations as well as detection of single pulses from known sources in dedicated VLBI observations. The underlying method takes the idea of geodetic VLBI data processing, including fringe fitting to maximize the signal power and geodetic VLBI solving for localization. By filtering the candidate signals with multiple windows within a baseline and by cross matching with multiple baselines, RFIs are eliminated effectively. Unlike the station…
In a search for short timescale astrophysical transients in time-domain data, radio-frequency interference (RFI) causes both large quantities of false positive candidates and a significant reduction in sensitivity if not correctly mitigated. Here we propose an algorithm that infers a time-variable frequency channel mask directly from short-duration ($\sim$1 s) data blocks: the method consists of computing a spectral statistic that correlates well with the presence of RFI, and then finding high outliers among the resulting values. For the latter task, we propose an outlier detection algorithm c…
Current and upcoming radio telescopes are being designed with increasing sensitivity to detect new and mysterious radio sources of astrophysical origin. While this increased sensitivity improves the likelihood of discoveries, it also makes these instruments more susceptible to the deleterious effects of Radio Frequency Interference (RFI). The challenge posed by RFI is exacerbated by the high data-rates achieved by modern radio telescopes, which require real-time processing to keep up with the data. Furthermore, the high data-rates do not allow for permanent storage of observations at high reso…
The Search for Extraterrestrial Intelligence (SETI) attempts to address the possibility of the presence of technological civilizations beyond the Earth. Benefiting from high sensitivity, large sky coverage, an innovative feed cabin for China's Five-hundred-meter Aperture Spherical radio Telescope (FAST), we performed the SETI first observations with FAST's newly commisioned 19-beam receiver; we report preliminary results in this paper. Using the data stream produced by the SERENDIP VI realtime multibeam SETI spectrometer installed at FAST, as well as its off-line data processing pipelines, we …
The Square Kilometre Array (SKA) will be both the largest radio telescope ever constructed and the largest Big Data project in the known Universe. The first phase of the project will generate on the order of 5 zettabytes of data per year. A critical task for the SKA will be its ability to process data for science, which will need to be conducted by science pipelines. Together with polarization data from the LOFAR Multifrequency Snapshot Sky Survey (MSSS), we have been developing a realistic SKA-like science pipeline that can handle the large data volumes generated by LOFAR at 150 MHz. The pipe…
Spiking Neural Networks for Radio Frequency Interference Detection in Radio Astronomy
International Centre for Radio Astronomy Research, University of Western Australia（西澳大利亚大学国际射电天文研究中心）；School of Physics, Mathematics and Computing, University of Western Australia（西澳大利亚大学物理、数学与计算学院）
Detection of radio emission from exoplanets can provide information on the star-planet system that is difficult to study otherwise, such as the planetary magnetic field, magnetosphere, rotation period, interior structure, atmospheric dynamics and escape, and any star-planet interactions. Such a detection in the radio domain would open up a whole new field in the study of exoplanets. However, currently there are no confirmed detections of an exoplanet at radio frequencies. In this study, we search for non-thermal radio emission from the 55 Cnc system which has 5 known exoplanets. According to t…
[abridged] The International LOFAR Telescope is an interferometer with stations spread across Europe. With baselines of up to ~2,000 km, LOFAR has the unique capability of achieving sub-arcsecond resolution at frequencies below 200 MHz, although this is technically and logistically challenging. Here we present a calibration strategy that builds on previous high-resolution work with LOFAR. We give an overview of the calibration strategy and discuss the special challenges inherent to enacting high-resolution imaging with LOFAR, and describe the pipeline, which is publicly available, in detail. W…
Radio frequency interference (RFI) detection and excision are key steps in the data-processing pipeline of the Five-hundred-meter Aperture Spherical radio Telescope (FAST). Because of its high sensitivity and large data rate, FAST requires more accurate and efficient RFI flagging methods than its counterparts. In the last decades, approaches based upon artificial intelligence (AI), such as codes using convolutional neural networks (CNNs), have been proposed to identify RFI more reliably and efficiently. However, RFI flagging of FAST data with such methods has often proved to be erroneous, with…
The Low Frequency Array (LOFAR) is an ideal instrument to conduct deep extragalactic surveys. It has a large field of view and is sensitive to large scale and compact emission. It is, however, very challenging to synthesize thermal noise limited maps at full resolution, mainly because of the complexity of the low-frequency sky and the direction dependent effects (phased array beams and ionosphere). In this first paper of a series we present a new calibration and imaging pipeline that aims at producing high fidelity, high dynamic range images with LOFAR High Band Antenna data, while being compu…
The LOw-Frequency ARray is a low-frequency radio interferometer composed by observational stations spread across Europe and it is the largest precursor of SKA in terms of effective area and generated data rates. In 2018, the Italian community officially joined LOFAR project, and it deployed a distributed computing and storage infrastructure dedicated to LOFAR data analysis. The infrastructure is based on 4 nodes distributed in different Italian locations and it offers services for pipelines execution, storage of final and intermediate results and support for the use of the software and infrast…
RFI-DRUnet: Restoring dynamic spectra corrupted by radio frequency interference -- Application to pulsar observations
CNRS（法国国家科学研究中心）；Institut de Recherche en Informatique de Toulouse (IRIT)（图卢兹计算机科学研究所（IRIT））；Toulouse INP（图卢兹国立综合理工学院）；University of Toulouse（图卢兹大学）；Station de Radioastronomie de Nançay（南赛射电天文台）；Observatoire de Paris（巴黎天文台）；PSL Research University（巴黎文理研究大学）；CNRS/INSU（法国国家科学研究中心/法国国家天文与地球物理研究所）；ENSEEIHT（国立高等电子、电子技术、计算机、水力与电信学院）
The Northern High Time Resolution Universe pulsar survey: II. Single-pulse search set-up and simulations
Radboud University（拉德堡德大学）；Max-Planck-Institut für Radioastronomie（马克斯·普朗克射电天文研究所）；ASTRON, the Netherlands Institute for Radio Astronomy（荷兰射电天文研究所）；University of Manchester（曼彻斯特大学）
An Attempt to Search for Unintended Electromagnetic Radiation from Starlink Satellites with the 21 Centimeter Array: Methodology and RFI Characterization
Guizhou Normal University（贵州师范大学）；Guizhou University（贵州大学）；Shanghai Astronomical Observatory, Chinese Academy of Sciences（中国科学院上海天文台）；University of Chinese Academy of Sciences（中国科学院大学）；ShanghaiTech University（上海科技大学）；National Astronomical Observatories, Chinese Academy of Sciences（中国科学院国家天文台）
We introduce a new pipeline for analyzing and mitigating radio frequency interference (RFI), which we call Sky-Subtracted Incoherent Noise Spectra (SSINS). SSINS is designed to identify and remove faint RFI below the single baseline thermal noise by employing a frequency-matched detection algorithm on baseline-averaged amplitudes of time-differenced visibilities. We demonstrate the capabilities of SSINS using the Murchison Widefield Array (MWA) in Western Australia. We successfully image aircraft flying over the array via digital television (DTV) reflection detected using SSINS and summarize a…
LOFAR is the only radio telescope that is presently capable of high-sensitivity, high-resolution (<1 mJy/b and <15") observations at ultra-low frequencies (<100 MHz). To utilise these capabilities, the LOFAR Surveys Key Science Project is undertaking a large survey to cover the entire northern sky with Low Band Antenna (LBA) observations. The LOFAR LBA Sky Survey (LoLSS) aims to cover the entire northern sky with 3170 pointings in the frequency range 42-66 MHz, at a resolution of 15 arcsec and at a sensitivity of 1 mJy/b. Here we outline the survey strategy, the observational status, the curre…
We conducted a search for technosignatures in April of 2018 and 2019 with the L-band receiver (1.15-1.73 GHz) of the 100 m diameter Green Bank Telescope. These observations focused on regions surrounding 31 Sun-like stars near the plane of the Galaxy. We present the results of our search for narrowband signals in this data set as well as improvements to our data processing pipeline. Specifically, we applied an improved candidate signal detection procedure that relies on the topographic prominence of the signal power, which nearly doubles the signal detection count of some previously analyzed d…
Trottier Space Institute, McGill University（特罗蒂尔太空研究所，麦吉尔大学）；Department of Physics, McGill University（麦吉尔大学物理系）；Anton Pannekoek Institute for Astronomy, University of Amsterdam（阿姆斯特丹大学安东·潘内科克天文学研究所）；ASTRON, Netherlands Institute for Radio Astronomy（ASTRON，荷兰射电天文研究所）
The radio-wavelength detection of extensive air showers (EAS) initiated by cosmic-ray interactions in the Earth's atmosphere is a promising technique for investigating the origin of these particles and the physics of their interactions. The Low Frequency Array (LOFAR) and the Owens Valley Long Wavelength Array (OVRO-LWA) have both demonstrated that the dense cores of low frequency radio telescope arrays yield detailed information on the radiation ground pattern, which can be used to reconstruct key EAS properties and infer the primary cosmic-ray composition. Here, we demonstrate a new observat…
As several large single-dish radio surveys begin operation within the coming decade, a wealth of radio data will become available and provide a new window to the Universe. In order to fully exploit the potential of these data sets, it is important to understand the systematic effects associated with the instrument and the analysis pipeline. A common approach to tackle this is to forward-model the entire system - from the hardware to the analysis of the data products. For this purpose, we introduce two newly developed, open-source Python packages: the HI Data Emulator (HIDE) and the Signal Extr…
The flux, polarimetric and spectral response of phased array radio telescopes with no moving parts such as LOFAR is known to vary considerably with orientation of the source to the receivers. Calibration models exist for this dependency such as those that are used in the LOFAR pipeline. Presented here is a system for comparing the predicted outputs from any given model with the results of an observation. In this paper, a sample observation of a bright source, Cassiopeia A, is used to demonstrate the software in operation, by providing an observation and a model of that observation which can be…
