集合虚拟机：一种多前端多后端数据分析的抽象
The Collection Virtual Machine: An Abstraction for Multi-Frontend Multi-Backend Data Analysis
摘要：
从数量不断增加的硬件平台中获得最佳性能一直是数据处理系统面临的一个反复出现的挑战。近年来，数据科学的出现及其越来越多和复杂的分析类型使这一挑战变得更加困难。在实践中，系统设计人员被组合的数量所淹没，通常只实现一个分析/平台组合，导致重复的实现工作-以及为数据科学家提供过多的半兼容工具。在本文中，我们提出了“集合虚拟机”（或CVM）-一个可扩展的编译器框架，旨在保持数据分析系统的专业化过程易于处理。它可以同时捕获大跨度的底层、特定于硬件的实现技术的本质，以及不同类型分析的高级操作。其核心是一种用于定义嵌套的、面向集合的中间表示（IR）的语言。前端以该语言定义的IR风格生成程序，通过一系列重写（可能多次更改IR风格）进行优化，直到程序最终以特定于平台的操作符的IR表示。在减少总体实施工作量的同时，这也提高了分析和硬件平台的互操作性。我们已经成功地使用CVM为多核CPU、RDMA集群和云中的无服务器计算基础设施等各种平台构建了专门的后端，并预计在不久的将来，更多的前端和硬件平台也会获得类似的结果。
英文摘要：
Getting the best performance from the ever-increasing number of hardware platforms has been a recurring challenge for data processing systems. In recent years, the advent of data science with its increasingly numerous and complex types of analytics has made this challenge even more difficult. In practice, system designers are overwhelmed by the number of combinations and typically implement only one analysis/platform combination, leading to repeated implementation effort -- and a plethora of semi-compatible tools for data scientists. In this paper, we propose the "Collection Virtual Machine" (or CVM) -- an extensible compiler framework designed to keep the specialization process of data analytics systems tractable. It can capture at the same time the essence of a large span of low-level, hardware-specific implementation techniques as well as high-level operations of different types of analyses. At its core lies a language for defining nested, collection-oriented intermediate representations (IRs). Frontends produce programs in their IR flavors defined in that language, which get optimized through a series of rewritings (possibly changing the IR flavor multiple times) until the program is finally expressed in an IR of platform-specific operators. While reducing the overall implementation effort, this also improves the interoperability of both analyses and hardware platforms. We have used CVM successfully to build specialized backends for platforms as diverse as multi-core CPUs, RDMA clusters, and serverless computing infrastructure in the cloud and expect similar results for many more frontends and hardware platforms in the near future.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
