通过数据扩散加速大规模数据探索
Accelerating Large-scale Data Exploration through Data Diffusion
摘要：
数据密集型应用程序通常需要对大型数据集进行探索性分析。如果在分布式资源上执行分析，数据局部性对于高吞吐量和性能至关重要。我们提出了一个“数据扩散”的方法，获取计算和存储资源动态，复制数据响应需求，并安排计算接近数据。随着需求的增加，会获得更多的资源，从而允许对引用相同数据的后续请求做出更快的响应;当需求下降时，资源会被释放。这种方法可以提供专用硬件的好处，而不会产生相关的高成本，具体取决于工作负载和资源特性。这种方法让人联想到协作缓存，Web缓存和对等存储系统，但解决了不同的应用需求。其他数据感知调度方法假设专用资源，如果负载显著变化，则这可能是昂贵的和/或低效的。为了探索数据扩散方法的可行性，我们扩展了Falkon资源提供和任务调度系统，以支持数据缓存和数据感知调度。微基准测试和大规模天文学应用程序的性能结果表明，我们的方法相对于替代方法提高了性能，并提供了更好的可扩展性，因为聚合I/O带宽随数据缓存节点的数量线性扩展。
英文摘要：
Data-intensive applications often require exploratory analysis of large datasets. If analysis is performed on distributed resources, data locality can be crucial to high throughput and performance. We propose a "data diffusion" approach that acquires compute and storage resources dynamically, replicates data in response to demand, and schedules computations close to data. As demand increases, more resources are acquired, thus allowing faster response to subsequent requests that refer to the same data; when demand drops, resources are released. This approach can provide the benefits of dedicated hardware without the associated high costs, depending on workload and resource characteristics. The approach is reminiscent of cooperative caching, web-caching, and peer-to-peer storage systems, but addresses different application demands. Other data-aware scheduling approaches assume dedicated resources, which can be expensive and/or inefficient if load varies significantly. To explore the feasibility of the data diffusion approach, we have extended the Falkon resource provisioning and task scheduling system to support data caching and data-aware scheduling. Performance results from both micro-benchmarks and a large scale astronomy application demonstrate that our approach improves performance relative to alternative approaches, as well as provides improved scalability as aggregated I/O bandwidth scales linearly with the number of data cache nodes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
