分布式计算中通过交织团的最优任务分配
Order Optimal Task Allocation in Distributed Computing via Interweaved Cliques
AI总结：
研究针对分布式计算系统中任务分配问题，提出交织团设计以优化通信和计算负载，实现通信成本与计算负载的最优缩放。
摘要：
我们考虑一个分布式计算系统，其中主节点协调N个工人评估一个函数，该函数接受一般分解。特别地，我们关注一般情况，即请求的函数允许d-均匀分解，意味着它可以分解为一组子函数，每个子函数依赖于n个文件中的唯一d元组。我们的目标是设计文件和任务分配，以最小化主节点到任何工人的最坏情况通信量以及工人之间的最坏情况计算负载。我们首先表明，具有最小通信和计算成本的最优文件和任务分配在组合设计理论中具有自然的特征化：它对应于Steiner系统S(t, k, v)，其中t=d，v=n，k≈n/N^{1/d}。然而，Steiner系统仅在非常受限的参数范围内存在。为了克服这一限制，我们提出了信息论启发的交织团（IC）设计，一种通用且确定性的分配框架，通过允许工人文件负载的微小变化来放松Steiner系统的严格结构。尽管略微次优，IC设计在通信成本上接近我们的下限，同时保持计算负载的最优缩放，从而使得该工作能够推导出该一般分布式计算问题的基本缩放定律，适用于广泛的参数范围。
英文摘要：
We consider a distributed computing system in which a master node coordinates $N$ workers to evaluate a function over $n$ input files, where this function accepts general decomposition. In particular, we focus on the general case where the requested function admits a $d$-uniform decomposition, meaning that it can be decomposed into a set of subfunctions that each depends on a unique $d$-tuple of the $n$ files. Our objective is to design file and task allocations that minimize the worst-case communication from the master to any worker and the worst-case computational load across workers. We first show that the optimal file and task allocation with minimum communication and computation costs admits a natural characterization within combinatorial design theory: it corresponds to a Steiner system $S(t, k, v)$ with $t=d$, $v=n$, and $k \approx \frac{n}{N^{1/d}}$. However, Steiner systems are known to exist only for very restricted parameter regimes. To overcome this limitation, we propose the information-theoretic-inspired \emph{Interweaved Clique (IC) design}, a universal and deterministic allocation framework that relaxes the strict structure of Steiner systems by allowing slight variations in worker file loads. Although slightly suboptimal, the IC design achieves a communication cost within a constant factor $4e$ from our converse, while also maintaining an order-optimal computation cost, thus allowing this work to derive the fundamental scaling laws of this general distributed computing problem for a large range of parameters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
