Towards Collaborative Optimization of Cluster Configurations for Distributed Dataflow Jobs
Jonathan Will, Jonathan Bader, Lauritz Thamsen
备注：
6 pages, 7 figures, 1 table; Associated experiment results: https://github.com/dos-group/c3o-experiments ; Appearence in the Proceedings of the 2020 IEEE International Conference on Big Data (Big Data); Presentation at the 4th International Workshop on Benchmarking, Performance Tuning and Optimization for Big Data Applications (BPOD). IEEE. 2020
Analyzing large datasets with distributed dataflow systems requires the use of clusters. Public cloud providers offer a large variety and quantity of resources that can be used for such clusters. However, picking the appropriate resources in both type and number can often be challenging, as the selected configuration needs to match a distributed dataflow job's resource demands and access patterns. A good cluster configuration avoids hardware bottlenecks and maximizes resource utilization, avoiding costly overprovisioning.
We propose a collaborative approach for finding optimal cluster configurations based on sharing and learning from historical runtime data of distributed dataflow jobs. Collaboratively shared data can be utilized to predict runtimes of future job executions through the use of specialized regression models. However, training prediction models on historical runtime data that were produced by different users and in diverse contexts requires the models to take these contexts into account.
FedACT: Concurrent Federated Intelligence across Heterogeneous Data Sources
School of Computing and Informatics, University of Louisiana at Lafayette, USA（路易斯安那州立大学拉斐特分校计算机与信息学院）；Department of Computer and Information Sciences, University of Delaware, USA（德雷塞尔大学计算机与信息科学系）；Department of Computer Science, University of Illinois at Urbana-Champaign, USA（伊利诺伊大学厄巴纳-香槟分校计算机科学系）
