HopsFS：使用NewSQL数据库扩展分层文件系统元数据
HopsFS: Scaling Hierarchical File System Metadata Using NewSQL Databases
摘要：
最近在无共享、事务性、内存中的NewSQL数据库的性能和可伸缩性方面的改进重新开启了一个研究问题，即分层文件系统的分布式元数据是否可以使用商品数据库进行管理。在本文中，我们介绍HopsFS，Hadoop分布式文件系统（HDFS）的下一代发行版，取代HDFS的单节点内存中的元数据服务，与分布式元数据服务建立在一个新的SQL数据库。通过消除元数据瓶颈，与HDFS相比，HopsFS支持更大数量级和更高吞吐量的集群。元数据容量已经增加到HDFS容量的至少37倍，并且在基于Spotify的工作负载跟踪的实验中，我们表明HopsFS支持Apache HDFS的16到37倍的吞吐量。HopsFS对于许多并发客户端也具有较低的延迟，并且在故障转移期间没有停机时间。最后，由于元数据现在存储在商品数据库中，因此可以安全地扩展并轻松导出到外部系统，以进行在线分析和自由文本搜索。
英文摘要：
Recent improvements in both the performance and scalability of shared-nothing, transactional, in-memory NewSQL databases have reopened the research question of whether distributed metadata for hierarchical file systems can be managed using commodity databases. In this paper, we introduce HopsFS, a next generation distribution of the Hadoop Distributed File System (HDFS) that replaces HDFS' single node in-memory metadata service, with a distributed metadata service built on a NewSQL database. By removing the metadata bottleneck, HopsFS enables an order of magnitude larger and higher throughput clusters compared to HDFS. Metadata capacity has been increased to at least 37 times HDFS' capacity, and in experiments based on a workload trace from Spotify, we show that HopsFS supports 16 to 37 times the throughput of Apache HDFS. HopsFS also has lower latency for many concurrent clients, and no downtime during failover. Finally, as metadata is now stored in a commodity database, it can be safely extended and easily exported to external systems for online analysis and free-text search.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
