通过有损数据压缩实现对快速瞬态事件的近实时远程搜索
Enabling near real-time remote search for fast transient events with lossy data compression
摘要:
我们提出了一个系统的评估JPEG 2000(ISO/IEC 15444)作为一种传输数据格式,使快速的瞬态事件的快速远程搜索的一部分,更深更广更快的程序(ESTA)。它使用了从无线电到伽马射线的20台望远镜,对毫秒到小时的时间尺度上的快速瞬变事件进行同步和快速响应的后续搜索。搜索需求有一组约束,这在大型协作中变得越来越普遍。在这里,我们专注于快速的光学数据组件,由暗能量相机(DECam)在CTIO。每个DECam图像共有70个CCD保存为约1.2 GB的FITS文件。近实时数据处理和快速瞬态候选识别-在其他望远镜上快速后续触发的几分钟内-需要超过CTIO现场现有的计算能力。在这种情况下,需要将数据文件快速传输到国外,以便进行超级计算后处理、源查找、可视化和分析。搜索过程中的这一步是一个主要的瓶颈,减少数据大小有助于适应更快的数据传输。为了最大限度地增加传输时间并实现我们的科学目标,我们选择了有损数据压缩-请记住,原始数据是存档的,可以在以后进行评估。我们评估有损JPEG 2000压缩如何影响寻找瞬变的过程中,发现只有一个可忽略不计的影响压缩比高达~25:1。我们还发现压缩比和平均估计的数据传输加速因子之间的线性关系。在科学管道中添加高度定制的压缩和解压缩步骤,大大减少了传输时间-验证了其对科学管道的引入,并使科学成为当前技术中的难点。
英文摘要:
We present a systematic evaluation of JPEG2000 (ISO/IEC 15444) as a transport data format to enable rapid remote searches for fast transient events as part of the Deeper Wider Faster program (DWF). DWF uses ~20 telescopes from radio to gamma-rays to perform simultaneous and rapid-response follow-up searches for fast transient events on millisecond-to-hours timescales. DWF search demands have a set of constraints that is becoming common amongst large collaborations. Here, we focus on the rapid optical data component of DWF led by the Dark Energy Camera (DECam) at CTIO. Each DECam image has 70 total CCDs saved as a ~1.2 gigabyte FITS file. Near real-time data processing and fast transient candidate identifications -- in minutes for rapid follow-up triggers on other telescopes -- requires computational power exceeding what is currently available on-site at CTIO. In this context, data files need to be transmitted rapidly to a foreign location for supercomputing post-processing, source finding, visualization and analysis. This step in the search process poses a major bottleneck, and reducing the data size helps accommodate faster data transmission. To maximise our gain in transfer time and still achieve our science goals, we opt for lossy data compression -- keeping in mind that raw data is archived and can be evaluated at a later time. We evaluate how lossy JPEG2000 compression affects the process of finding transients, and find only a negligible effect for compression ratios up to ~25:1. We also find a linear relation between compression ratio and the mean estimated data transmission speed-up factor. Adding highly customized compression and decompression steps to the science pipeline considerably reduces the transmission time -- validating its introduction to the DWF science pipeline and enabling science that was otherwise too difficult with current technology.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。