arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2107.08594cs.DBcs.LG
收藏

无服务器查询的最佳资源分配

Optimal Resource Allocation for Serverless Queries

Microsoft(微软);Two Sigma(双西格玛)

Anish Pimpley, Shuo Li, Anubha Srivastava, Vishal Rohra, Yi Zhu, Soundararajan Srinivasan, Alekh Jindal, Hiren Patel, Shi Qiao, Rathijit Sen

摘要:

优化分析工作负载的资源分配对于降低云数据服务的成本至关重要。与此同时,在无服务器处理系统中,用户很难为每个查询分配资源,并且他们经常以数量级错误分配。不幸的是,以前的工作集中在预测峰值分配,而忽略了积极的权衡资源分配和运行时间。此外,这些方法无法预测过去未观察到的查询的分配。在本文中,我们解决这两个问题。我们引入了一个系统的最佳资源分配,可以预测性能与积极的权衡,新的和过去观察到的查询。我们引入了性能特征曲线(PCC)的概念,作为一个参数化的表示,可以compensately捕捉资源和性能之间的关系。为了解决训练数据的稀疏性,我们引入了一种新的数据增强技术,使用单次查询来有效地合成整个PCC。最后,我们展示了与GNN相结合的约束损失函数的优势,与传统的ML方法相比,通过对微软SCOPE大数据工作负载的广泛实验评估来捕获特定于域的行为。

英文摘要:

Optimizing resource allocation for analytical workloads is vital for reducing costs of cloud-data services. At the same time, it is incredibly hard for users to allocate resources per query in serverless processing systems, and they frequently misallocate by orders of magnitude. Unfortunately, prior work focused on predicting peak allocation while ignoring aggressive trade-offs between resource allocation and run-time. Additionally, these methods fail to predict allocation for queries that have not been observed in the past. In this paper, we tackle both these problems. We introduce a system for optimal resource allocation that can predict performance with aggressive trade-offs, for both new and past observed queries. We introduce the notion of a performance characteristic curve (PCC) as a parameterized representation that can compactly capture the relationship between resources and performance. To tackle training data sparsity, we introduce a novel data augmentation technique to efficiently synthesize the entire PCC using a single run of the query. Lastly, we demonstrate the advantages of a constrained loss function coupled with GNNs, over traditional ML methods, for capturing the domain specific behavior through an extensive experimental evaluation over SCOPE big data workloads at Microsoft.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑