ShoreKeeper:通过分布式恢复成本感知缓存加速云中的迭代数据分析
ShoreKeeper: Accelerating Iterative Data Analytics in the Cloud via Distributed Recovery-Cost-Aware Caching
AI总结:
ShoreKeeper是将集群内存视为统一弹性池的框架,通过全局优化模型和图剪枝技术降低数据恢复成本,在各类云实例上运行,性能优于现有最先进系统。
摘要:
由迭代图处理或机器学习处理驱动的大规模数据密集型工作负载,由于不断生成新的中间数据,对内存有着无法满足的需求。尽管现有的缓存策略力求将性能关键数据保留在内存中,但在实际应用中往往失效,因为它们局限于节点本地的内存管理,且假设节点可用性较高。本文提出ShoreKeeper,这是一个将集群范围内存视为统一弹性池以最大化执行效率的框架。ShoreKeeper采用全局优化模型,将分布式数据流映射为单一图抽象,以最小化集群中从缓存驱逐数据的恢复成本。为应对内存工作集和集群容量随时间波动的情况,ShoreKeeper采用图剪枝技术,允许快速优化缓存计划。这些优化使ShoreKeeper无论是部署在高可用虚拟机(VM)还是成本效益高的可抢占实例上,都能无缝运行。实验表明,ShoreKeeper始终优于最先进的系统。
英文摘要:
Large-scale data-intensive workloads powered by iterative graph or machine learning processing have an insatiable demand for memory due to the continuous generation of new intermediate data. Although existing caching strategies strive to keep performance-critical data in memory, they often fail in practice because they are limited to node-local memory management and assume high node availability. In this paper, we introduce ShoreKeeper, a framework that maximizes execution efficiency by treating cluster-wide memory as a unified, elastic pool. ShoreKeeper employs a global optimization model that maps distributed dataflows into a single graph abstraction to minimize the recovery cost of data evicted from cache across the cluster. To handle memory working sets and cluster capacities that fluctuate over time, ShoreKeeper employs a graph-pruning technique that permits the fast refinement of the caching plan. These optimizations allow ShoreKeeper to work seamlessly whether deployed on highly available VMs or cost-effective preemptible instances in the cloud. Our experiments show that ShoreKeeper consistently outperforms state-of-the-art systems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。