一个高效、平衡的数据并行子采样工作负载平台
An Efficient and Balanced Platform for Data-Parallel Subsampling Workloads
摘要:
随着互联网服务的出现,数据的增长速度开始超过其处理速度。为了个性化用户体验,必须以交互方式实时的处理这些庞大的数据。为了实现更快的数据处理,往往采用一种称为二次抽样的统计方法.子采样工作负载从样本数据的随机子集(即,子样本)。数据并行平台将这些样本分组到任务中;每个任务并行地对其数据进行子采样。目前,Hadoop等最先进的平台是为长时间运行的大型任务而构建的,但是平均任务大小较小的应用程序在这些平台上会承受很大的开销。二次采样工作负载中的任务的大小可以最大限度地减少总体缓存未命中的数量,并且这些任务可以在几秒钟内完成。这种技术可以减少映射缩减作业的总长度,但只有当减少缓存未命中率所节省的时间不会被任务创建和数据分发的平台开销所抵消时。在这篇论文中,我们提出了一个数据并行平台,一个有效的数据分布组件,打破数据并行子采样工作负载到计算集群与微小的任务。每个微小的任务在几百毫秒到几秒内完成。小任务减少了随机二次采样导致的处理器缓存未命中,从而加快了每个任务的运行时间。然而,它们会导致显著的调度开销和数据分发挑战。我们提出了一个任务拐点算法和一个动态调度器,调度任务的工作节点的可用性和响应时间的数据节点的基础上。我们将我们的框架与BashReduce和Hadoop的各种配置进行比较。详细讨论了两个工作负载,EAGLET和Netflix的电影评级的微小任务的方法。
英文摘要:
With the advent of internet services, data started growing faster than it can be processed. To personalize user experience, this enormous data has to be processed in real time, in interactive fashion. In order to achieve faster data processing often a statistical method called subsampling. Subsampling workloads compute statistics from a random subset of sample data (i.e., a subsample). Data-parallel platforms group these samples into tasks; each task subsamples its data in parallel. Current, state-of-the-art platforms such as Hadoop are built for large tasks that run for long periods of time, but applications with smaller average task sizes suffer large overheads on these platforms. Tasks in subsampling workloads are sized to minimize the number of overall cache misses, and these tasks can complete in seconds. This technique can reduce the overall length of a map-reduce job, but only when the savings from the cache miss rate reduction are not eclipsed by the platform overhead of task creation and data distribution. In this thesis, we propose a data-parallel platform with an efficient data distribution component that breaks data-parallel subsampling workloads into compute clusters with tiny tasks. Each tiny task completes in few hundreds of milliseconds to seconds. Tiny tasks reduce processor cache misses caused by random subsampling, which speeds up per-task running time. However, they cause significant scheduling overheads and data distribution challenges. We propose a task knee-pointing algorithm and a dynamic scheduler that schedules the tasks to worker nodes based on the availability and response times of the data nodes. We compare our framework against various configurations of BashReduce and Hadoop. A detailed discussion of tiny task approach on two workloads, EAGLET and Netflix movie rating is presented.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。