MapReduce中不同规模输入的分配问题
Assignment Problems of Different-Sized Inputs in MapReduce
摘要：
MapReduce算法可以通过映射模式来描述，该映射模式将输入分配给一组归约器，使得对于每个所需的输出，存在接收参与该输出的计算的所有输入的归约器。归约器有一个容量，它限制了可以分配给它们的输入集。然而，各个输入可能在大小方面有所不同。我们认为，第一次，映射模式的输入大小的考虑和限制的一部分。在任何MapReduce作业中需要优化的重要参数之一是map和reduce阶段之间的通信成本。通信成本可以通过最小化发送到归约器的输入的副本的数量来优化。通信成本与用于适当地容纳输入的受限容量的减速器的数量密切相关，从而满足输入必须在减速器中满足的要求。在这项工作中，我们考虑一个家庭的问题，它需要每个输入满足彼此的输入在至少一个减速器。我们还考虑了一个稍微不同的问题族，其中，列表X的每个输入都需要满足另一个列表Y的每个输入，至少在一个reducer中。我们证明了寻找这些问题族的最佳映射方案是NP-困难的，并提出了一个基于装箱的近似算法寻找一个近最佳的映射方案。
英文摘要：
A MapReduce algorithm can be described by a mapping schema, which assigns inputs to a set of reducers, such that for each required output there exists a reducer that receives all the inputs that participate in the computation of this output. Reducers have a capacity, which limits the sets of inputs that they can be assigned. However, individual inputs may vary in terms of size. We consider, for the first time, mapping schemas where input sizes are part of the considerations and restrictions. One of the significant parameters to optimize in any MapReduce job is communication cost between the map and reduce phases. The communication cost can be optimized by minimizing the number of copies of inputs sent to the reducers. The communication cost is closely related to the number of reducers of constrained capacity that are used to accommodate appropriately the inputs, so that the requirement of how the inputs must meet in a reducer is satisfied. In this work, we consider a family of problems where it is required that each input meets with each other input in at least one reducer. We also consider a slightly different family of problems in which, each input of a list, X, is required to meet each input of another list, Y, in at least one reducer. We prove that finding an optimal mapping schema for these families of problems is NP-hard, and present a bin-packing-based approximation algorithm for finding a near optimal mapping schema.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
