天文学探索性数据分析的最小重计算
Minimal Re-computation for Exploratory Data Analysis in Astronomy
摘要：
我们提出了一种技术，用于在迭代更改或添加数据分析程序时自动最大限度地减少重新计算，这在天文学探索性数据分析中经常出现。一个典型的例子是标记和校准的要求或不寻常的观察，其中目视检查表明改进的处理策略。该技术是基于memoization和引用透明的任务。我们描述了这种技术的CASA射电天文数据简化包的实施。我们还提出了一种技术，用于优化存储效率的memoized中间数据产品使用写时复制和块级重复数据删除和测量其实际效率。我们发现最小重新计算技术提高了数据分析的效率，同时减少了用户错误的可能性，提高了最终结果的再现性。它还有助于在批调度集群计算机系统上进行探索性数据分析。
英文摘要：
We present a technique to automatically minimise the re-computation when a data analysis program is iteratively changed, or added to, as is often the case in exploratory data analysis in astronomy. A typical example is flagging and calibration of demanding or unusual observations where visual inspection suggests improvement to the processing strategy. The technique is based on memoization and referentially transparent tasks. We describe the implementation of this technique for the CASA radio astronomy data reduction package. We also propose a technique for optimising efficiency of storage of memoized intermediate data products using copy-on-write and block level de-duplication and measure their practical efficiency. We find the minimal recomputation technique improves the efficiency of data analysis while reducing the possibility for user error and improving the reproducibility of the final result. It also aids exploratory data analysis on batch-schedule cluster computer systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
