This paper considers the MapReduce-like coded distributed computing framework originally proposed by Li et al., which uses coding techniques when distributed computing servers exchange their computed intermediate values, in order to reduce the overall traffic load. Their original model servers are connected via an error-free common communication bus allowing broadcast transmissions. However, this assumption is one of the major limitations in practice since the practical cloud computing network topologies are far more involved than a simple single bus. We formulate a topological coded distributed computing problem, where the distributed servers communicate with each other through some switch network. By using a special instance of fat-tree topologies, referred to as t-ary fat-tree proposed by Al-Fares et al. which can be built by some cheap switches, we propose a coded distributed computing scheme to achieve the minimum max-link communication load defined as the maximum load over all links.
This paper is concerned with linear superposition systems in which all components of the superimposed signal are coded with an identical binary low-density parity-check (LDPC) code.
Content delivery networks store information distributed across multiple servers, so as to balance the load and avoid unrecoverable losses in case of node or disk failures. Coded caching has been shown to be a useful technique which can reduce peak traffic rates by pre-fetching popular content at the end users and encoding transmissions so that different users can extract different information from the same packet. On one hand, distributed storage limits the capability of combining content from different servers into a single message, causing performance losses in coded caching schemes. But, on…
Distributed computing platforms typically assume the availability of reliable and dedicated connections among the processors. This work considers an alternative scenario, relevant for wireless data centers and federated learning, in which the distributed processors, operating on generally distinct coded data, are connected via shared wireless channels accessed via full-duplex transmission. The study accounts for both wireless and computing impairments, including interference, imperfect Channel State Information, and straggling processors, and it assumes a Map-Shuffle-Reduce coded computing par…
In this paper, we investigate network nest coded modulation schemes for multiple access relay channels. The performance of the distributed systems which are based on distributed convolutional codes with network coded modulation is presented. An analytical upper bound on bit error probability performance for the studied distributed systems with Maximum Likelihood Sequence Detection (MLSD) is derived. The constructured bounds for the investigated systems are shown to be asymptotically tight for increasing channel Signal-to-Noise Ratio (SNR) values
In this paper, we consider a distributed reception scenario where a transmitter broadcasts a signal to multiple geographically separated receive nodes over fading channels, and each node forwards a few bits representing a processed version of the received signal to a fusion center. The fusion center then tries to decode the transmitted signal based on the forwarded information from the receive nodes and possible channel state information. We show that there is a strong connection between the problem of minimizing a symbol error probability at the fusion center in distributed reception and chan…
Matrix factorization is an important representation learning algorithm, e.g., recommender systems, where a large matrix can be factorized into the product of two low dimensional matrices termed as latent representations. This paper investigates the problem of matrix factorization in distributed computing systems with stragglers, those compute nodes that are slow to return computation results. A computation procedure, called coded Alternative Least Square (ALS), is proposed for mitigating the effect of stragglers in such systems. The coded ALS algorithm iteratively computes two low dimensional …
We address the use of maximum distance separable (MDS) codes for distributed storage (DS) to enable efficient content delivery in wireless networks. Content is stored in a number of the mobile devices and can be retrieved from them using device-to-device communication or, alternatively, from the base station (BS). We derive an analytical expression for the download delay in the hypothesis that the reliability state of the network is periodically restored. Our analysis shows that MDS-coded DS can dramatically reduce the download time with respect to the reference scenario where content is alway…
Li {\it et al}. introduced coded distributed computing (CDC) scheme to reduce the communication load in general distributed computing frameworks such as MapReduce. They also proposed cascaded CDC schemes where each output function is computed multiple times, and proved that such schemes achieved the fundamental trade-off between computation load and communication load. However, these schemes require exponentially large numbers of input files and output functions when the number of computing nodes gets large. In this paper, by using the structure of placement delivery arrays (PDAs), we construc…
In this paper, we propose a new rateless coded cooperation scheme for a general multi-user cooperative wireless system. We develop cooperation methods based on Raptor codes with the assumption that the channels face erasure with specific erasure probabilities and transmitters have no channel state information. A fully coded cooperation (FCC) and a partially coded cooperation (PCC) strategy are developed to maximize the average system throughput. Both PCC and FCC schemes have been analyzed through AND-OR tree analysis and a linear programming optimization problem is then formulated to find the …
We consider the recently proposed Coded Distributed Computing (CDC) framework that leverages carefully designed redundant computations to enable coding opportunities that substantially reduce the communication load of distributed computing. We generalize this framework to heterogeneous systems where different nodes in the computing cluster can have different storage (or processing) capabilities. We provide the information-theoretically optimal data set placement and coded data shuffling scheme that minimizes the communication load in a cluster with 3 nodes. For clusters with $K>3$ nodes, we pr…
Coded distributed computing (CDC) introduced by Li et al. in 2015 offers an efficient approach to trade computing power to reduce the communication load in general distributed computing frameworks such as MapReduce. For the more general cascaded CDC, Map computations are repeated at $r$ nodes to significantly reduce the communication load among nodes tasked with computing $Q$ Reduce functions $s$ times. While an achievable cascaded CDC scheme was proposed, it only operates on homogeneous networks, where the storage, computation load and communication load of each computing node is the same. In…
Cache-aided coded multicast leverages side information at wireless edge caches to efficiently serve multiple groupcast demands via common multicast transmissions, leading to load reductions that are proportional to the aggregate cache size. However, the increasingly unpredictable and personalized nature of the content that users consume challenges the efficiency of existing caching-based solutions in which only exact content reuse is explored. This paper generalizes the cache-aided coded multicast problem to a source compression with distributed side information problem that specifically accou…
Coded computation is a framework which provides redundancy in distributed computing systems to speed up largescale tasks. Although most existing works assume an error-free scenarios in a master-worker setup, the link failures are common in current wired/wireless networks. In this paper, we consider the straggler problem in coded distributed computing with link failures, by modeling the links between the master node and worker nodes as packet erasure channels. When the master fails to detect the received signal, retransmission is required for each worker which increases the overall run-time to …
We consider the problem of coded distributed computing where a large linear computational job, such as a matrix multiplication, is divided into $k$ smaller tasks, encoded using an $(n,k)$ linear code, and performed over $n$ distributed nodes. The goal is to reduce the average execution time of the computational job. We provide a connection between the problem of characterizing the average execution time of a coded distributed computing system and the problem of analyzing the error probability of codes of length $n$ used over erasure channels. Accordingly, we present closed-form expressions for…
Distributed computation is a framework used to break down a complex computational task into smaller tasks and distributing them among computational nodes. Erasure correction codes have recently been introduced and have become a popular workaround to the well known ``straggling nodes'' problem, in particular, by matching linear coding for linear computation tasks. It was observed that decoding tends to amplify the computation ``noise'', i.e., the numerical errors at the computation nodes. We propose taking advantage of the case that more nodes return than minimally required. We show how a cleve…
Several network communication problems are highly related such as coded caching and distributed computation. The centralized coded caching focuses on reducing the network burden in peak times in a wireless network system and the coded distributed computation studies the tradeoff between computation and communication in distributed system. In this paper, motivated by the study of the only rainbow $3$-term arithmetic progressions set, we propose a unified framework for constructing coded caching schemes. This framework builds bridges between coded caching schemes and lots of combinatorial object…
Coded distributed computing framework enables large-scale machine learning (ML) models to be trained efficiently in a distributed manner, while mitigating the straggler effect. In this work, we consider a multi-task assignment problem in a coded distributed computing system, where multiple masters, each with a different matrix multiplication task, assign computation tasks to workers with heterogeneous computing capabilities. Both dedicated and probabilistic worker assignment models are considered, with the objective of minimizing the average completion time of all computations. For dedicated w…
We consider private information retrieval (PIR) for distributed storage systems (DSSs) with noncolluding nodes where data is stored using a non maximum distance separable (MDS) linear code. It was recently shown that if data is stored using a particular class of non-MDS linear codes, the MDS-PIR capacity, i.e., the maximum possible PIR rate for MDS-coded DSSs, can be achieved. For this class of codes, we prove that the PIR capacity is indeed equal to the MDS-PIR capacity, giving the first family of non-MDS codes for which the PIR capacity is known. For other codes, we provide asymmetric PIR pr…
An adaptive joint power allocation (JPA) and linear receiver design algorithm using the minimum bit error rate (MBER) criterion for a cooperative Multiple-Input Multiple-Output (MIMO) network is proposed. The system employs multiple relays with Distributed Space-Time Coding (DSTC) schemes and an Amplify-and-Forward (AF) strategy. It is designed according to a joint constrained optimization algorithm to determine the MBER power allocation parameters and the receive filter parameters for each transmitted symbol. The simulation results indicate that the proposed algorithm obtains performance gain…
Elasticity is offered by cloud service providers to exploit under-utilized computing resources. The low-cost elastic nodes can leave and join any time during the computation cycle. The possibility of elastic events occurring together with the problem of slow nodes, referred to as stragglers, increases the uncertainty of the system, leading to computation delay. Recent results have shown that coded computing can be used to reduce the negative effect of elasticity and stragglers. In this paper, we propose two hierarchical coded elastic computing schemes that can further speed up the system by ex…
Edge caching and computing have been regarded as an efficient approach to tackle the wireless spectrum crunch problem. In this paper, we design a general coded caching with device computing strategy for content computation, e.g., virtual reality (VR) rendering, to minimize the average transmission bandwidth with the caching capacity and the energy constraints of each mobile device, and the maximum tolerable delay constraint of each task. The key enabler is that because both coded data and stored data can be the data before or after computing, the proposed scheme has numerous edge computing and…
In this paper, we propose a new coded computing technique called "substitute decoding" for general iterative distributed computation tasks. In the first part of the paper, we use PageRank as a simple example to show that substitute decoding can make the computation of power iterations solving PageRank on sparse matrices robust to erasures in distributed systems. For these sparse matrices, codes with dense generator matrices can significantly increase storage costs and codes with low-density generator matrices (LDGM) are preferred. Surprisingly, we show through both theoretical analysis and sim…
We investigate the use of maximum distance separable (MDS) codes to cache popular content to reduce the download delay of wireless content delivery. In particular, we consider a cellular system where devices roam in an out of a cell according to a Poisson random process. Popular content is cached in a limited number of the mobile devices using an MDS code and can be downloaded from the mobile devices using device-to-device communication. We derive an analytical expression for the delay incurred in downloading content from the wireless network and show that distributed caching using MDS codes c…
We consider a mobile edge computing problem, in which mobile users offload their computation tasks to computing nodes (e.g., base stations) at the network edge. The edge nodes compute the requested functions and communicate the computed results to the users via wireless links. For this problem, we propose a Universal Coded Edge Computing (UCEC) scheme for linear functions to simultaneously minimize the load of computation at the edge nodes, and maximize the physical-layer communication efficiency towards the mobile users. In the proposed UCEC scheme, edge nodes create coded inputs of the users…
This work explores a distributed computing setting where $K$ nodes are assigned fractions (subtasks) of a computational task in order to perform the computation in parallel. In this setting, a well-known main bottleneck has been the inter-node communication cost required to parallelize the task, because unlike the computational cost which could keep decreasing as $K$ increases, the communication cost remains approximately constant, thus bounding the total speedup gains associated to having more computing nodes. This bottleneck was substantially ameliorated by the recent introduction of coded M…
In this paper, we consider a hierarchical distributed multi-task learning (MTL) system where distributed users wish to jointly learn different models orchestrated by a central server with the help of a layer of multiple relays. Since the users need to download different learning models in the downlink transmission, the distributed MTL suffers more severely from the communication bottleneck compared to the single-task learning system. To address this issue, we propose a coded hierarchical MTL scheme that exploits the connection topology and introduces coding techniques to reduce communication l…
This paper investigates distributed computing systems where computations are split into "Map" and "Reduce" functions. A new coded scheme, called distributed computing and coded communication (D3C), is proposed, and its communication load is analyzed as a function of the available storage space and the number of intermediate values (IVA) to be computed. D3C achieves the smallest possible communication load for a given storage space, while a smaller number of IVAs need to be computed compared to Li et al.'s coded distributed computing (CDC) scheme. More generally, our scheme can flexibly trade b…
Coded distributed computing (CDC) introduced by Li et. al. is an effective technique to trade computation load for communication load in a MapReduce framework. CDC achieves an optimal trade-off by duplicating map computations at $r$ computing nodes to yield multicasting opportunities such that $r$ nodes are served simultaneously in the Shuffle phase. However, in general, the state-of-the-art CDC scheme is mainly designed only for homogeneous networks, where the computing nodes are assumed to have the same storage, computation and communication capabilities. In this work, we explore two novel a…
Consider a multi-cell mobile edge computing network, in which each user wishes to compute the product of a user-generated data matrix with a network-stored matrix. This is done through task offloading by means of input uploading, distributed computing at edge nodes (ENs), and output downloading. Task offloading may suffer long delay since servers at some ENs may be straggling due to random computation time, and wireless channels may experience severe fading and interference. This paper aims to investigate the interplay among upload, computation, and download latencies during the offloading pro…
Matrix multiplication is a fundamental building block for large scale computations arising in various applications, including machine learning. There has been significant recent interest in using coding to speed up distributed matrix multiplication, that are robust to stragglers (i.e., machines that may perform slower computations). In many scenarios, instead of exact computation, approximate matrix multiplication, i.e., allowing for a tolerable error is also sufficient. Such approximate schemes make use of randomization techniques to speed up the computation process. In this paper, we initiat…
We consider a MapReduce-type task running in a distributed computing model which consists of ${K}$ edge computing nodes distributed across the edge of the network and a Master node that assists the edge nodes to compute output functions. The Master node and the edge nodes, both equipped with some storage memories and computing capabilities, are connected through a multicast network. We define the communication time spent during the transmission for the sequential implementation (all nodes send symbols sequentially) and parallel implementation (the Master node can send symbols during the edge n…
Cloud providers have recently introduced new offerings whereby spare computing resources are accessible at discounts compared to on-demand computing. Exploiting such opportunity is challenging inasmuch as such resources are accessed with low-priority and therefore can elastically leave (through preemption) and join the computation at any time. In this paper, we design a new technique called coded elastic computing, enabling distributed computations over elastic resources. The proposed technique allows machines to leave the computation without sacrificing the algorithm-level performance, and, a…
We consider distributed and dynamic caching of coded content at small base stations (SBSs) in an area served by a macro base station (MBS). Specifically, content is encoded using a maximum distance separable code and cached according to a time-to-live (TTL) cache eviction policy, which allows coded packets to be removed from the caches at periodic times. Mobile users requesting a particular content download coded packets from SBSs within communication range. If additional packets are required to decode the file, these are downloaded from the MBS. We formulate an optimization problem that is ef…
Coded distributed computing has been considered as a promising technique which makes large-scale systems robust to the "straggler" workers. Yet, practical system models for distributed computing have not been available that reflect the clustered or grouped structure of real-world computing servers. Neither the large variations in the computing power and bandwidth capabilities across different servers have been properly modeled. We suggest a group-based model to reflect practical conditions and develop an appropriate coding scheme for this model. The suggested code, called group code, employs p…
In distributed computing systems slow working nodes, known as stragglers, can greatly extend finishing times. Coded computing is a technique that enables straggler-resistant computation. Most coded computing techniques presented to date provide robustness by ensuring that the time to finish depends only on a set of the fastest nodes. However, while stragglers do compute less work than non-stragglers, in real-world commercial cloud computing systems (e.g., Amazon's Elastic Compute Cloud (EC2)) the distinction is often a soft one. In this paper, we develop hierarchical coded computing that explo…
The growing size of modern datasets necessitates splitting a large scale computation into smaller computations and operate in a distributed manner. Adversaries in a distributed system deliberately send erroneous data in order to affect the computation for their benefit. Boolean functions are the key components of many applications, e.g., verification functions in blockchain systems and design of cryptographic algorithms. We consider the problem of computing a Boolean function in a distributed computing system with particular focus on \emph{security against Byzantine workers}. Any Boolean funct…
The uplink of a Cloud Radio Access Network (CRAN) architecture is studied, where decoding at the cloud takes place at distributed decoding processors. To mitigate the impact of straggling decoders in the cloud, the cloud re-encodes the received frames via a linear code before distributing them to the decoding processors. Focusing on Gaussian channels, and assuming the use of lattice codes at the users, in this paper the maximum user rate is derived such that all the servers can reliably recover the linear combinations of the messages corresponding to the employed linear code at the cloud. Furt…
Communication overhead is one of the major performance bottlenecks in large-scale distributed computing systems, in particular for machine learning applications. Conventionally, compression techniques are used to reduce the load of communication by combining intermediate results of the same computation task as much as possible. Recently, via the development of coded distributed computing (CDC), it has been shown that it is possible to enable coding opportunities across intermediate results of different computation tasks to further reduce the communication load. We propose a new scheme, named c…
We consider the problem of computing the convolution of two long vectors using parallel processing units in the presence of "stragglers". Stragglers refer to the small fraction of faulty or slow processors that delays the entire computation in time-critical distributed systems. We first show that splitting the vectors into smaller pieces and using a linear code to encode these pieces provides better resilience against stragglers than replication-based schemes under a simple, worst-case straggler analysis. We then demonstrate that under commonly used models of computation time, coding can drama…
In large scale distributed computing systems, communication overhead is one of the major bottlenecks. In the map-shuffle-reduce framework, which is one of the major distributed computing frameworks, the communication load among servers can be reduced by increasing the computation load of each server, that is, there is a trade-off between computation load and communication load. Recently, it has been shown that coded distributed computing (CDC) improves this trade-off relationship by letting servers encode their intermediate computation results. The original CDC scheme does not assume any speci…
Coded distributed batch computation distributes a computation task, such as matrix multiplication, $N$-linear computation, or multivariate polynomial evaluation, across $S$ servers through a coding scheme, such that the response from any $R$ servers ($R$ is called the recovery threshold) is sufficient for the user to recover the desired computed value. Current approaches are based on either exclusively matrix-partitioning (Entangled Polynomial (EP) Codes for matrix multiplication), or exclusively batch processing (Lagrange Coded Computing (LCC)). We present three related classes of codes, base…
We study the numerical stability of polynomial based encoding methods, which has emerged to be a powerful class of techniques for providing straggler and fault tolerance in the area of coded computing. Our contributions are as follows: 1) We construct new codes for matrix multiplication that achieve the same fault/straggler tolerance as the previously constructed MatDot Codes and Polynomial Codes. Unlike previous codes that use polynomials expanded in a monomial basis, our codes uses a basis of orthogonal polynomials. 2) We show that the condition number of every $m \times m$ sub-matrix of an …
In this paper, a joint power allocation algorithm with minimum mean-squared error (MMSE) receiver for a cooperative Multiple-Input and Multiple-Output (MIMO) network which employs multiple relays and a Decode-and-Forward (DF) strategy is proposed. A Distributed Space-Time Coding (DSTC) scheme is applied in each relay node. We present a joint constrained optimization algorithm to determine the power allocation parameters and the MMSE receive filter parameter vectors for each transmitted symbol in each link, as well as the channel coefficients matrix. A Stochastic Gradient (SG) algorithm is deri…
University of Michigan（密歇根大学）；University of Southern California（南加州大学）
We consider the problem of coded computing, where a computational task is performed in a distributed fashion in the presence of adversarial workers. We propose techniques to break the adversarial toleration threshold barrier previously known in coded computing. More specifically, we leverage list-decoding techniques for folded Reed-Solomon codes and propose novel algorithms to recover the correct codeword using side information. In the coded computing setting, we show how the master node can perform certain carefully designed extra computations to obtain the side information. The workload of c…
