面向列的MapReduce存储技术
Column-Oriented Storage Techniques for MapReduce
摘要：
MapReduce的用户在扩展工作负载时经常会遇到性能问题。他们遇到的许多问题都可以通过应用从三十多年的并行DBMS研究中学到的技术来克服。然而，将这些技术转换为MapReduce实现（如Hadoop）带来了独特的挑战，可能会导致新的设计选择。本文描述了如何将面向列的存储技术以保留其流行编程API的方式并入Hadoop中。我们表明，在Hadoop中简单地使用二进制存储格式可以提供3倍的性能提升，而不是简单地使用文本文件。然后，我们介绍了一种面向列的存储格式，它与Hadoop的复制和调度约束兼容，并表明它可以在实际工作负载上加速MapReduce作业。我们还表明，处理复杂的列类型，如数组，映射和嵌套记录，这是常见的MapReduce作业，可以产生显着的CPU开销。最后，我们引入了一种新颖的跳表列格式和惰性记录构造策略，避免了重复不需要的记录，从而提供了额外的1.5倍性能提升。一个真正的内联网爬行的实验表明，我们的面向列的存储技术可以提高Hadoop中的地图阶段的性能多达两个数量级。
英文摘要：
Users of MapReduce often run into performance problems when they scale up their workloads. Many of the problems they encounter can be overcome by applying techniques learned from over three decades of research on parallel DBMSs. However, translating these techniques to a MapReduce implementation such as Hadoop presents unique challenges that can lead to new design choices. This paper describes how column-oriented storage techniques can be incorporated in Hadoop in a way that preserves its popular programming APIs. We show that simply using binary storage formats in Hadoop can provide a 3x performance boost over the naive use of text files. We then introduce a column-oriented storage format that is compatible with the replication and scheduling constraints of Hadoop and show that it can speed up MapReduce jobs on real workloads by an order of magnitude. We also show that dealing with complex column types such as arrays, maps, and nested records, which are common in MapReduce jobs, can incur significant CPU overhead. Finally, we introduce a novel skip list column format and lazy record construction strategy that avoids deserializing unwanted records to provide an additional 1.5x performance boost. Experiments on a real intranet crawl are used to show that our column-oriented storage techniques can improve the performance of the map phase in Hadoop by as much as two orders of magnitude.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
