Coded computing is an effective technique to mitigate "stragglers" in large-scale and distributed matrix multiplication. In particular, univariate polynomial codes have been shown to be effective in straggler mitigation by making the computation time depend only on the fastest workers. However, these schemes completely ignore the work done by the straggling workers resulting in a waste of computational resources. To reduce the amount of work left unfinished at workers, one can further decompose the matrix multiplication task into smaller sub-tasks, and assign multiple sub-tasks to each worker, possibly heterogeneously, to better fit their particular storage and computation capacities. In this work, we propose a novel family of bivariate polynomial codes to efficiently exploit the work carried out by straggling workers. We show that bivariate polynomial codes bring significant advantages in terms of upload communication costs and storage efficiency, measured in terms of the number of sub-tasks that can be computed per worker. We propose two bivariate polynomial coding schemes. The first one exploits the fact that bivariate interpolation is always possible on a rectangular grid of evaluation points. We obtain such points at the cost of adding some redundant computations. For the second scheme, we relax the decoding constraints and require decodability for almost all choices of the evaluation points. We present interpolation sets satisfying such decodability conditions for certain storage configurations of workers. Our numerical results show that bivariate polynomial coding considerably reduces the average computation time of distributed matrix multiplication. We believe this work opens up a new class of previously unexplored coding schemes for efficient coded distributed computation.
We consider the problem of private distributed matrix multiplication under limited resources. Coded computation has been shown to be an effective solution in distributed matrix multiplication, both providing privacy against the workers and boosting the computation speed by efficiently mitigating stragglers. In this work, we propose the use of recently-introduced bivariate polynomial codes to further speed up private distributed matrix multiplication by exploiting the partial work done by the stragglers rather than completely ignoring them. We show that the proposed approach reduces the average…
We consider the problem of secure distributed matrix multiplication (SDMM). Coded computation has been shown to be an effective solution in distributed matrix multiplication, both providing privacy against workers and boosting the computation speed by efficiently mitigating stragglers. In this work, we present a non-direct secure extension of the recently introduced bivariate polynomial codes. Bivariate polynomial codes have been shown to be able to further speed up distributed matrix multiplication by exploiting the partial work done by the stragglers rather than completely ignoring them whil…
In this paper, delay-free, low complexity, joint source-channel coding (JSCC) for transmission of two correlated Gaussian memoryless sources over a Gaussian Multiple Access Channel (GMAC) is considered. The main contributions of the paper are two distributed JSCC schemes: one discrete scheme based on nested scalar quantization, and one hybrid discrete-analog scheme based on a scalar quantizer and a linear continuous mapping. The proposed schemes show promising performance which improve with increasing correlation and are robust against variations in noise level. Both schemes exhibit a constant…
Coded distributed computing was recently introduced to mitigate the effect of stragglers on distributed computing. This paper combines ideas of approximate computing with coded computing to further accelerate computation. We propose successive approximation coding (SAC) techniques that realize a tradeoff between accuracy and speed, allowing the distributed computing system to produce approximations that increase in accuracy over time. If a sufficient number of compute nodes finish their tasks, SAC exactly recovers the desired computation. We theoretically provide design guidelines for our SAC …
Edge computing is emerging as a new paradigm to allow processing data at the edge of the network, where data is typically generated and collected, by exploiting multiple devices at the edge collectively. However, exploiting the potential of edge computing is challenging mainly due to the heterogeneous and time-varying nature of edge devices. Coded computation, which advocates mixing data in sub-tasks by employing erasure codes and offloading these sub-tasks to other devices for computation, is recently gaining interest, thanks to its higher reliability, smaller delay, and lower communication c…
Starting with the multiplication of elements in $\mathbb{F}_{q}^2$ which is consistent with that over $\mathbb{F}_{q^2}$, where $q$ is a prime power, via some identification of the two environments, we investigate the $c$-differential uniformity for bivariate functions $F(x,y)=(G(x,y),H(x,y))$. By carefully choosing the functions $G(x,y)$ and $H(x,y)$, we present several constructions of bivariate functions with low $c$-differential uniformity. Many P$c$N and AP$c$N functions can be produced from our constructions.
We consider the problem of massive matrix multiplication, which underlies many data analytic applications, in a large-scale distributed system comprising a group of worker nodes. We target the stragglers' delay performance bottleneck, which is due to the unpredictable latency in waiting for slowest nodes (or stragglers) to finish their tasks. We propose a novel coding strategy, named \emph{entangled polynomial code}, for designing the intermediate computations at the worker nodes in order to minimize the recovery threshold (i.e., the number of workers that we need to wait for in order to compu…
Distributed matrix multiplication is widely used in several scientific domains. It is well recognized that computation times on distributed clusters are often dominated by the slowest workers (called stragglers). Recent work has demonstrated that straggler mitigation can be viewed as a problem of designing erasure codes. For matrices $\mathbf A$ and $\mathbf B$, the technique essentially maps the computation of $\mathbf A^T \mathbf B$ into the multiplication of smaller (coded) submatrices. The stragglers are treated as erasures in this process. The computation can be completed as long as a cer…
Recent years have seen renewed attention to arithmetic coding (AC). This is thanks to the use of AC for distribution matching (DM) to control the channel input distribution in probabilistic amplitude shaping. There are two main problems inherent to AC: (1) its required arithmetic precision grows linearly with the input length, and (2) high-precision multiplications and divisions are required. Here, we introduce a multiplication-free AC-based DM technique via three lookup tables (LUTs) which solves both problems above. These LUTs are used to approximate the high-precision multiplications and di…
We provide novel coded computation strategies for distributed matrix-matrix products that outperform the recent "Polynomial code" constructions in recovery threshold, i.e., the required number of successful workers. When $m$-th fraction of each matrix can be stored in each worker node, Polynomial codes require $m^2$ successful workers, while our MatDot codes only require $2m-1$ successful workers, albeit at a higher communication cost from each worker to the fusion node. We also provide a systematic construction of MatDot codes. Further, we propose "PolyDot" coding that interpolates between Po…
In this paper, we propose CodedSketch, as a distributed straggler-resistant scheme to compute an approximation of the multiplication of two massive matrices. The objective is to reduce the recovery threshold, defined as the total number of worker nodes that we need to wait for to be able to recover the final result. To exploit the fact that only an approximated result is required, in reducing the recovery threshold, some sorts of pre-compression are required. However, compression inherently involves some randomness that would lose the structure of the matrices. On the other hand, considering t…
We provide a complete characterization of the achievable distortion region for the problem of sending a bivariate Gaussian source over bandwidth-matched Gaussian broadcast channels, where each receiver is interested in only one component of the source. This setting naturally generalizes the simple single Gaussian source bandwidth-matched broadcast problem for which the uncoded scheme is known to be optimal. We show that a hybrid scheme can achieve the optimum for the bivariate case, but neither an uncoded scheme alone nor a separation-based scheme alone is sufficient. We further show that in t…
Large-scale machine learning and data mining applications require computer systems to perform massive matrix-vector and matrix-matrix multiplication operations that need to be parallelized across multiple nodes. The presence of straggling nodes -- computing nodes that unpredictably slowdown or fail -- is a major bottleneck in such distributed computations. Ideal load balancing strategies that dynamically allocate more tasks to faster nodes require knowledge or monitoring of node speeds as well as the ability to quickly move data. Recently proposed fixed-rate erasure coding strategies can handl…
The complexity of software implementations of MDS erasure codes mainly depends on the efficiency of the finite field operations implementation. In this paper, we propose a method to reduce the complexity of the finite field multiplication by using fast transforms between a field and a ring to perform the multiplication in a ring. We show that moving to a ring reduces the complexity of the operations. Then, we show that this construction allows the use of simple scheduling to reduce the number of operations.
We propose two coding schemes for distributed matrix multiplication in the presence of stragglers. These coding schemes are adaptations of LT codes and Raptor codes to distributed matrix multiplication and are termed \emph{factored LT (FLT) codes} and \emph{factored Raptor (FR) codes}. Empirically, we show that FLT codes have near-optimal recovery thresholds when the number of worker nodes is very large, and that FR codes have excellent recovery thresholds while the number of worker nodes is moderately large. FLT and FR codes have better recovery thresholds when compared to Product codes and t…
Matrix multiplication is a fundamental building block for large scale computations arising in various applications, including machine learning. There has been significant recent interest in using coding to speed up distributed matrix multiplication, that are robust to stragglers (i.e., machines that may perform slower computations). In many scenarios, instead of exact computation, approximate matrix multiplication, i.e., allowing for a tolerable error is also sufficient. Such approximate schemes make use of randomization techniques to speed up the computation process. In this paper, we initiat…
Large-scale machine learning and data mining methods routinely distribute computations across multiple agents to parallelize processing. The time required for computation at the agents is affected by the availability of local resources giving rise to the "straggler problem" in which the computation results are held back by unresponsive agents. For this problem, linear coding of the matrix sub-blocks can be used to introduce resilience toward straggling. The Parameter Server (PS) utilizes a channel code and distributes the matrices to the workers for multiplication. It then produces an approxim…
Slow working nodes, known as stragglers, can greatly reduce the speed of distributed computation. Coded matrix multiplication is a recently introduced technique that enables straggler-resistant distributed multiplication of large matrices. A key property is that the finishing time depends only on the work completed by a set of the fastest workers, while the work done by the slowest workers is ignored completely. This paper is motivated by the observation that in real-world commercial cloud computing systems such as Amazon's Elastic Compute Cloud (EC2) the distinction between fast and slow node…
Coded distributed computing has been considered as a promising technique which makes large-scale systems robust to the "straggler" workers. Yet, practical system models for distributed computing have not been available that reflect the clustered or grouped structure of real-world computing servers. Neither the large variations in the computing power and bandwidth capabilities across different servers have been properly modeled. We suggest a group-based model to reflect practical conditions and develop an appropriate coding scheme for this model. The suggested code, called group code, employs p…
We introduce two generalizations to the paradigm of using Random Khatri-Rao Product (RKRP) codes for distributed matrix multiplication. We first introduce a class of codes called Sparse Random Khatri-Rao Product (SRKRP) codes which have sparse generator matrices. SRKRP codes result in lower encoding, computation and communication costs than RKRP codes when the input matrices are sparse, while they exhibit similar numerical stability to other state of the art schemes. We empirically study the relationship between the probability of the generator matrix (restricted to the set of non-stragglers) …
Coded matrix multiplication is a technique to enable straggler-resistant multiplication of large matrices in distributed computing systems. In this paper, we first present a conceptual framework to represent the division of work amongst processors in coded matrix multiplication as a cuboid partitioning problem. This framework allows us to unify existing methods and motivates new techniques. Building on this framework, we apply the idea of hierarchical coding (Ferdinand & Draper, 2018) to coded matrix multiplication. The hierarchical scheme we develop is able to exploit the work completed by al…
We study coded distributed matrix multiplication from an approximate recovery viewpoint. We consider a system of $P$ computation nodes where each node stores $1/m$ of each multiplicand via linear encoding. Our main result shows that the matrix product can be recovered with $ε$ relative error from any $m$ of the $P$ nodes for any $ε> 0$. We obtain this result through a careful specialization of MatDot codes -- a class of matrix multiplication codes previously developed in the context of exact recovery ($ε=0$). Since prior results showed that MatDot codes achieve the best exact recovery threshol…
In this work we have considered formal power series and partial differential equations, and their relationship with Coding Theory. We have obtained the nature of solutions for the partial differential equations for Cycle Poisson Case. The coefficients for this case have been simulated, and the high tendency of growth is shown. In the light of Complex Analysis, the Hadamard Multiplication's Theorem is presented as a new approach to divide the power sums relating to the error probability, each part of which can be analyzed later.
This paper considers fast algorithms for operations on linearized polynomials. We propose a new multiplication algorithm for skew polynomials (a generalization of linearized polynomials) which has sub-quadratic complexity in the polynomial degree $s$, independent of the underlying field extension degree~$m$. We show that our multiplication algorithm is faster than all known ones when $s \leq m$. Using a result by Caruso and Le Borgne (2017), this immediately implies a sub-quadratic division algorithm for linearized polynomials for arbitrary polynomial degree $s$. Also, we propose algorithms wi…
A novel implementation of a special class of Galois ring, in which the multiplication can be realized by a cyclic convolution, is applied to the construction of network codes. The primitive operations involved are byte-wise shifts and integer additions modulo a power of 2. Both of them can be executed efficiently in microprocessors. An illustration of how to apply this idea to array code is given at the end of the paper.
We consider a large-scale matrix multiplication problem where the computation is carried out using a distributed system with a master node and multiple worker nodes, where each worker can store parts of the input matrices. We propose a computation strategy that leverages ideas from coding theory to design intermediate computations at the worker nodes, in order to efficiently deal with straggling workers. The proposed strategy, named as \emph{polynomial codes}, achieves the optimum recovery threshold, defined as the minimum number of workers that the master needs to wait for in order to compute…
Differentially Private Secure Multiplication: Hiding Information in the Rubble of Noise
Georgia Institute of Technology（佐治亚理工学院）；Pennsylvania State University（宾夕法尼亚州立大学）；University of California, Santa Barbara（加州大学圣巴巴拉分校）；Harvard University（哈佛大学）
Coded computation is an emerging research area that leverages concepts from erasure coding to mitigate the effect of stragglers (slow nodes) in distributed computation clusters, especially for matrix computation problems. In this work, we present a class of distributed matrix-vector multiplication schemes that are based on codes in the Rosenbloom-Tsfasman metric and universally decodable matrices. Our schemes take into account the inherent computation order within a worker node. In particular, they allow us to effectively leverage partial computations performed by stragglers (a feature that ma…
This paper has two contributions. First, we propose a novel coded matrix multiplication technique called Generalized PolyDot codes that advances on existing methods for coded matrix multiplication under storage and communication constraints. This technique uses "garbage alignment," i.e., aligning computations in coded computing that are not a part of the desired output. Generalized PolyDot codes bridge between Polynomial codes and MatDot codes, trading off between recovery threshold and communication costs. Second, we demonstrate that Generalized PolyDot can be used for training large Deep Neu…
Coding schemes are presented that provide the ability to correct and detect computational errors while using dot-product engines for integer vector--matrix multiplication. Both the $L_1$-metric and the Hamming metric are considered.
A secure multi-party batch matrix multiplication problem (SMBMM) is considered, where the goal is to allow a master to efficiently compute the pairwise products of two batches of massive matrices, by distributing the computation across S servers. Any X colluding servers gain no information about the input, and the master gains no additional information about the input beyond the product. A solution called Generalized Cross Subspace Alignment codes with Noise Alignment (GCSA-NA) is proposed in this work, based on cross-subspace alignment codes. The state of art solution to SMBMM is a coding sch…
Large-scale machine learning and data mining methods routinely distribute computations across multiple agents to parallelize processing. The time required for the computations at the agents is affected by the availability of local resources and/or poor channel conditions giving rise to the "straggler problem". As a remedy to this problem, we employ Unequal Error Protection (UEP) codes to obtain an approximation of the matrix product in the distributed computation setting to provide higher protection for the blocks with higher effect on the final result. We characterize the performance of the p…
We consider the problem of designing a coding scheme that allows both sparsity and privacy for distributed matrix-vector multiplication. Perfect information-theoretic privacy requires encoding the input sparse matrices into matrices distributed uniformly at random from the considered alphabet; thus destroying the sparsity. Computing matrix-vector multiplication for sparse matrices is known to be fast. Distributing the computation over the non-sparse encoded matrices maintains privacy, but introduces artificial computing delays. In this work, we relax the privacy constraint and show that a cert…
In wireless broadcast, random linear network coding (RLNC) over GF(2^L) is known to asymptotically achieve the optimal completion delay with increasing L. However, the high decoding complexity hinders the potential applicability of RLNC schemes over large GF(2^L). In this paper, a comprehensive analysis of completion delay and decoding complexity is conducted for field-based systematic RLNC schemes in wireless broadcast. In particular, we prove that the RLNC scheme over GF(2) can also asymptotically approach the optimal completion delay per packet when the packet number goes to infinity. Moreo…
Distributed matrix computations -- matrix-matrix or matrix-vector multiplications -- are well-recognized to suffer from the problem of stragglers (slow or failed worker nodes). Much of prior work in this area is (i) either sub-optimal in terms of its straggler resilience, or (ii) suffers from numerical problems, i.e., there is a blow-up of round-off errors in the decoded result owing to the high condition numbers of the corresponding decoding matrices. Our work presents convolutional coding approach to this problem that removes these limitations. It is optimal in terms of its straggler resilie…
Large matrix multiplications are central to large-scale machine learning applications. These operations are often carried out on a distributed computing platform with a master server and multiple workers in the cloud operating in parallel. For such distributed platforms, it has been recently shown that coding over the input data matrices can reduce the computational delay, yielding a trade-off between recovery threshold, i.e., the number of workers required to recover the matrix product, and communication load, i.e., the total amount of data to be downloaded from the workers. In this paper, in…
In large-scale distributed computing clusters, such as Amazon EC2, there are several types of "system noise" that can result in major degradation of performance: bottlenecks due to limited communication bandwidth, latency due to straggler nodes, etc. On the other hand, these systems enjoy abundance of redundancy - a vast number of computing nodes and large storage capacity. There have been recent results that demonstrate the impact of coding for efficient utilization of computation and storage redundancy to alleviate the effect of stragglers and communication bottlenecks in homogeneous cluster…
In secure distributed matrix multiplication (SDMM) the multiplication $\mathbf{A}\mathbf{B}$ from two private matrices $\mathbf{A}$ and $\mathbf{B}$ is outsourced by a user to $N$ distributed servers. In $\ell$-SDMM, the goal is to a design a joint communication-computation procedure that optimally balances conflicting communication and computation metrics without leaking any information on both $\mathbf{A}$ and $\mathbf{B}$ to any set of $\ell\leq N$ servers. To this end, the user applies coding with $\tilde{\mathbf{A}}_i$ and $\tilde{\mathbf{B}}_i$ representing encoded versions of $\mathbf{A…
Skew polynomials are a class of non-commutative polynomials that have several applications in computer science, coding theory and cryptography. In particular, skew polynomials can be used to construct and decode evaluation codes in several metrics, like e.g. the Hamming, rank, sum-rank and skew metric. We propose a fast divide-and-conquer variant of Kötter-Nielsen-Høholdt (KNH) interpolation algorithm: it inputs a list of linear functionals on skew polynomial vectors, and outputs a reduced Gröbner basis of their kernel intersection. We show, that the proposed KNH interpolation can be used to s…
Coded distributed batch computation distributes a computation task, such as matrix multiplication, $N$-linear computation, or multivariate polynomial evaluation, across $S$ servers through a coding scheme, such that the response from any $R$ servers ($R$ is called the recovery threshold) is sufficient for the user to recover the desired computed value. Current approaches are based on either exclusively matrix-partitioning (Entangled Polynomial (EP) Codes for matrix multiplication), or exclusively batch processing (Lagrange Coded Computing (LCC)). We present three related classes of codes, base…
Coded computation is a method to mitigate "stragglers" in distributed computing systems through the use of error correction coding that has lately received significant attention. First used in vector-matrix multiplication, the range of application was later extended to include matrix-matrix multiplication, heterogeneous networks, convolution, and approximate computing. A drawback to previous results is they completely ignore work completed by stragglers. While stragglers are slower compute nodes, in many settings the amount of work completed by stragglers can be non-negligible. Thus, in this w…
Tensor operations, such as matrix multiplication, are central to large-scale machine learning applications. For user-driven tasks these operations can be carried out on a distributed computing platform with a master server at the user side and multiple workers in the cloud operating in parallel. For distributed platforms, it has been recently shown that coding over the input data matrices can reduce the computational delay, yielding a trade-off between recovery threshold and communication load. In this paper we impose an additional security constraint on the data matrices and assume that worke…
Building on the previous work of Lee et al. and Ferdinand et al. on coded computation, we propose a sequential approximation framework for solving optimization problems in a distributed manner. In a distributed computation system, latency caused by individual processors ("stragglers") usually causes a significant delay in the overall process. The proposed method is powered by a sequential computation scheme, which is designed specifically for systems with stragglers. This scheme has the desirable property that the user is guaranteed to receive useful (approximate) computation results whenever …
Inexpensive cloud services, such as serverless computing, are often vulnerable to straggling nodes that increase end-to-end latency for distributed computation. We propose and implement simple yet principled approaches for straggler mitigation in serverless systems for matrix multiplication and evaluate them on several common applications from machine learning and high-performance computing. The proposed schemes are inspired by error-correcting codes and employ parallel encoding and decoding over the data stored in the cloud using serverless workers. This creates a fully distributed computing …
In cloud computing systems slow processing nodes, often referred to as "stragglers", can significantly extend the computation time. Recent results have shown that error correction coding can be used to reduce the effect of stragglers. In this work we introduce a scheme that, in addition to using error correction to distribute mixed jobs across nodes, is also able to exploit the work completed by all nodes, including stragglers. We first consider vector-matrix multiplication and apply maximum distance separable (MDS) codes to small blocks of sub-matrices. The worker nodes process blocks sequent…
While performing distributed computations in today's cloud-based platforms, execution speed variations among compute nodes can significantly reduce the performance and create bottlenecks like stragglers. Coded computation techniques leverage coding theory to inject computational redundancy and mitigate stragglers in distributed computations. In this paper, we propose a dynamic workload distribution strategy for coded computation called Slack Squeeze Coded Computation ($S^2C^2$). $S^2C^2$ squeezes the compute slack (i.e., overhead) that is built into the coded computing frameworks by efficientl…
