arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2006.03206cs.DCcs.DB
收藏

在大于内存的存储中实现高吞吐量和弹性

Achieving High Throughput and Elasticity in a Larger-than-Memory Store

Chinmay Kulkarni, Badrish Chandramouli, Ryan Stutsman

发表信息：
PVLDB, 14(8): 1427 - 1440, 2021
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

摘要：

数以百万计的传感器、移动应用程序和机器现在产生数十亿个事件。如果事件是在同一台机器上生成的，则专用的众核键值存储（KVS）可以以高速率（在一台机器上超过100 Mops/s）摄取和索引这些事件;然而，为了实用和经济高效，它们必须通过网络摄取事件并跨云资源弹性扩展。我们介绍了Shadowfax，一个基于FASTER的新的分布式KVS，它透明地跨越DRAM，SSD和云blob存储，同时使用传统的Linux TCP在商业Azure VM上提供130 Mops/s/VM。除了高单VM性能之外，Shadowfax还使用了一种独特的分布式重新配置方法，可以避免在正常操作和迁移期间进行任何服务器端密钥所有权检查或跨核心协调。因此，Shadowfax可以在17秒内转移负载，将系统吞吐量提高10 Mops/s，并且几乎没有中断。与最先进的技术相比，它的吞吐量提高了8倍（比Seastar+memcached），并避免了在迁移过程中移动冷数据的昂贵I/O。在12台机器上，Shadowfax保持其高吞吐量，执行930 Mops/s，据我们所知，这是用于大规模数据摄取和索引的分布式KVS的最高吞吐量。

英文摘要：

Millions of sensors, mobile applications and machines now generate billions of events. Specialized many-core key-value stores (KVSs) can ingest and index these events at high rates (over 100 Mops/s on one machine) if events are generated on the same machine; however, to be practical and cost-effective they must ingest events over the network and scale across cloud resources elastically. We present Shadowfax, a new distributed KVS based on FASTER, that transparently spans DRAM, SSDs, and cloud blob storage while serving 130 Mops/s/VM over commodity Azure VMs using conventional Linux TCP. Beyond high single-VM performance, Shadowfax uses a unique approach to distributed reconfiguration that avoids any server-side key ownership checks or cross-core coordination both during normal operation and migration. Hence, Shadowfax can shift load in 17 s to improve system throughput by 10 Mops/s with little disruption. Compared to the state-of-the-art, it has 8x better throughput (than Seastar+memcached) and avoids costly I/O to move cold data during migration. On 12 machines, Shadowfax retains its high throughput to perform 930 Mops/s, which, to the best of our knowledge, is the highest reported throughput for a distributed KVS used for large-scale data ingestion and indexing.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑