Apache Arrow Flight的基准测试--用于数据传输、查询和微服务的线速协议
Benchmarking Apache Arrow Flight -- A wire-speed protocol for data transfer, querying and microservices
摘要：
在不同的大数据框架和/或数据仓库/存储系统之间移动结构化数据通常会导致显著的开销。大多数情况下，访问数据所花费的总时间的80%以上都是在序列化/反序列化步骤中花费的。列式数据格式在分析数据库和事务数据库中越来越流行。Apache Arrow是一种统一的列式内存数据格式，它承诺提供高效的数据存储、访问、操作和传输。此外，通过引入构建在gRPC之上的Arrow Flight通信功能，Arrow可以通过TCP网络实现高性能数据传输。Arrow Flight允许以平台和语言独立的方式通过网络进行并行Arrow RecordBatch传输，并基于开源标准提供高性能，并行性和安全性。在本文中，我们汇集了一些最近实现的Arrow Flight用例及其基准测试结果。这些用例包括批量Arrow数据传输，查询子系统和Flight作为微服务集成到不同的框架中，以显示该协议的吞吐量和可扩展性结果。我们表明，飞行是能够实现高达6000 MB/s和4800 MB/s的吞吐量分别为DoGet（）和DoPut（）操作。在Mellanox ConnectX-3或Connect-IB互连节点上，Flight可以利用高达95%的总可用带宽。Flight是可扩展的，可以有效地使用多达一半的可用系统核心进行双向通信。对于像Dremio这样的查询系统，Flight比ODBC和turbodbc协议快一个数量级。与turbodbc和ODBC连接相比，Dremio上基于Arrow Flight的实现的性能分别提高了20倍和30倍。
英文摘要：
Moving structured data between different big data frameworks and/or data warehouses/storage systems often cause significant overhead. Most of the time more than 80\% of the total time spent in accessing data is elapsed in serialization/de-serialization step. Columnar data formats are gaining popularity in both analytics and transactional databases. Apache Arrow, a unified columnar in-memory data format promises to provide efficient data storage, access, manipulation and transport. In addition, with the introduction of the Arrow Flight communication capabilities, which is built on top of gRPC, Arrow enables high performance data transfer over TCP networks. Arrow Flight allows parallel Arrow RecordBatch transfer over networks in a platform and language-independent way, and offers high performance, parallelism and security based on open-source standards. In this paper, we bring together some recently implemented use cases of Arrow Flight with their benchmarking results. These use cases include bulk Arrow data transfer, querying subsystems and Flight as a microservice integration into different frameworks to show the throughput and scalability results of this protocol. We show that Flight is able to achieve up to 6000 MB/s and 4800 MB/s throughput for DoGet() and DoPut() operations respectively. On Mellanox ConnectX-3 or Connect-IB interconnect nodes Flight can utilize upto 95\% of the total available bandwidth. Flight is scalable and can use upto half of the available system cores efficiently for a bidirectional communication. For query systems like Dremio, Flight is order of magnitude faster than ODBC and turbodbc protocols. Arrow Flight based implementation on Dremio performs 20x and 30x better as compared to turbodbc and ODBC connections respectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
