This paper focuses on mitigating the impact of stragglers in distributed learning system. Unlike the existing results designed for a fixed number of stragglers, we developed a new scheme called Adaptive Gradient Coding(AGC) with flexible tolerance of various number of stragglers. Our scheme gives an optimal tradeoff between computation load, straggler tolerance and communication cost. In particular, it allows to minimize the communication cost according to the real-time number of stragglers in the practical environments. Implementations on Amazon EC2 clusters using Python with mpi4py package verify the flexibility in several situations.
An adaptive joint source-channel coding (JSCC) scheme is presented for transmitting correlated sources over discrete-memoryless two-way channels subject to distortion constraints. The proposed JSCC scheme makes use of the previously transmitted and received channel signals as well as the sources' correlation to facilitate coordination between terminals. It is shown that the adaptive scheme strictly subsumes prior lossy coding methods for two-way simultaneous transmission and yields a new adaptive separate source-channel coding result. Two examples are given to show the scheme's advantages.
A subspace-based approach for rate-adaptive distributed source coding (DSC) based on discrete Fourier transform (DFT) codes is developed. Punctured DFT codes can be used to implement rate-adaptive source coding, however they perform poorly after even moderate puncturing since the performance of the subspace error localization degrades severely. The proposed subspace-based error localization extends and improves the existing one, based on additional syndrome, and is naturally suitable for rate-adaptive distributed source coding architecture.
A common complaint about adaptive prefix coding is that it is much slower than static prefix coding. Karpinski and Nekrich recently took an important step towards resolving this: they gave an adaptive Shannon coding algorithm that encodes each character in (O (1)) amortized time and decodes it in (O (\log H)) amortized time, where $H$ is the empirical entropy of the input string $s$. For comparison, Gagie's adaptive Shannon coder and both Knuth's and Vitter's adaptive Huffman coders all use (Θ(H)) amortized time for each character. In this paper we give an adaptive Shannon coder that both enco…
In this paper, we propose novel energy efficient adaptive network coding and modulation schemes for time variant channels. We evaluate such schemes under a realistic channel model for open area environments and Geostationary Earth Orbit (GEO) satellites. Compared to non-adaptive network coding and adaptive rate efficient network-coded schemes for time variant channels, we show that our proposed schemes, through physical layer awareness can be designed to transmit only if a target quality of service (QoS) is achieved. As a result, such schemes can provide remarkable energy savings.
A major hurdle in machine learning is scalability to massive datasets. Approaches to overcome this hurdle include compression of the data matrix and distributing the computations. \textit{Leverage score sampling} provides a compressed approximation of a data matrix using an importance weighted subset. \textit{Gradient coding} has been recently proposed in distributed optimization to compute the gradient using multiple unreliable worker nodes. By designing coding matrices, gradient coded computations can be made resilient to stragglers, which are nodes in a distributed network that degrade syst…
In this paper, we propose two novel physical layer aware adaptive network coding and coded modulation schemes for time variant channels. The proposed schemes have been applied to different satellite communications scenarios with different Round Trip Times (RTT). Compared to adaptive network coding, and classical non-adaptive network coding schemes for time variant channels, as benchmarks, the proposed schemes demonstrate that adaptation of packet transmission based on the channel variation and corresponding erasures allows for significant gains in terms of throughput, delay and energy efficien…
This paper considers the problem of source coding with side information at the decoder, also called Slepian-Wolf source coding scheme. In practical applications of this coding scheme, the statistical relation between the source and the side information can vary from one data transmission to another, and there is a need to adapt the coding rate depending on the current statistical relation. In this paper, we propose a novel rate-adaptive code construction based on LDPC codes for the Slepian-Wolf source coding scheme. The proposed code design method allows to optimize the code degree distributio…
The analysis of modulation schemes for the physical layer network-coded two way relaying scenario is presented which employs two phases: Multiple access (MA) phase and Broadcast (BC) phase. It was shown by Koike-Akino et. al. that adaptively changing the network coding map used at the relay according to the channel conditions greatly reduces the impact of multiple access interference which occurs at the relay during the MA phase. Depending on the signal set used at the end nodes, deep fades occur for a finite number of channel fade states referred as the singular fade states. The singular fade…
We consider distributed gradient descent in the presence of stragglers. Recent work on \em gradient coding \em and \em approximate gradient coding \em have shown how to add redundancy in distributed gradient descent to guarantee convergence even if some workers are \em stragglers\em---that is, slow or non-responsive. In this work we propose an approximate gradient coding scheme called \em Stochastic Gradient Coding \em (SGC), which works when the stragglers are random. SGC distributes data points redundantly to workers according to a pair-wise balanced design, and then simply ignores the strag…
An adaptive randomized distributed space-time coding (DSTC) scheme and algorithms are proposed for two-hop cooperative MIMO networks. Linear minimum mean square error (MMSE) receivers and an amplify-and-forward (AF) cooperation strategy are considered. In the proposed DSTC scheme, a randomized matrix obtained by a feedback channel is employed to transform the space-time coded matrix at the relay node. Linear MMSE expressions are devised to compute the parameters of the adaptive randomized matrix and the linear receive filter. A stochastic gradient algorithm is also developed to compute the par…
In this paper, we investigate the impact of coding on the Age of Information (AoI) in a two-user broadcast symbol erasure channel with feedback. We assume each update consists of $K$ symbols and the source is able to broadcast one symbol in each time slot. Due to random channel noise, the intended symbol at each user will be erased according to an independent and identically distributed (i.i.d.) Bernoulli process. A user is able to successfully decode an update if it accumulates sufficient information and successfully decodes the $K$ symbols of the update. Assuming prefect feedback information…
We propose an adaptive lossy joint source-channel coding (JSCC) scheme for sending correlated sources over two-terminal discrete-memoryless two-way channels (DM-TWCs). The main idea is to couple the independent operations of the terminals via an adaptive coding mechanism, which can mitigate cross-interference resulting from simultaneous channel transmissions and concurrently exploit the sources' correlation to reduce the end-to-end reconstruction distortions. Our adaptive JSCC scheme not only subsumes existing lossy coding methods for two-way simultaneous communication but also improves their …
The problem of predicting a sequence $x_1,x_2,...$ generated by a discrete source with unknown statistics is considered. Each letter $x_{t+1}$ is predicted using information on the word $x_1x_2... x_t$ only. In fact, this problem is a classical problem which has received much attention. Its history can be traced back to Laplace. We address the problem where each $x_i$ belongs to some large (or even infinite) alphabet. A method is presented for which the precision is greater than for known algorithms, where precision is estimated by the Kullback-Leibler divergence. The results can readily be tr…
An adaptive delay-tolerant distributed space-time coding (DSTC) scheme that exploits feedback is proposed for two-hop cooperative MIMO networks. Maximum likelihood (ML) receivers and adjustable code matrices are considered subject to a power constraint with a decode-and-forward (DF) cooperation strategy. In the proposed delay-tolerant DSTC scheme, an adjustable code matrix is employed to transform the space-time coded matrix at the relay nodes. Least-squares (LS) algorithms are developed with reduced computational complexity to adjust the parameters of the codes. Simulation results show that t…
Gradient coding is a technique for straggler mitigation in distributed learning. In this paper we design novel gradient codes using tools from classical coding theory, namely, cyclic MDS codes, which compare favorably with existing solutions, both in the applicable range of parameters and in the complexity of the involved algorithms. Second, we introduce an approximate variant of the gradient coding problem, in which we settle for approximate gradient computation instead of the exact one. This approach enables graceful degradation, i.e., the $\ell_2$ error of the approximate gradient is a decr…
We consider a wireless communication network with an adaptive scheme to select the number of packets to be admitted and encoded for each transmission, and characterize the information timeliness. For a network of erasure channels and discrete time, we provide closed form expressions for the Average and Peak Age of Information (AoI) as functions of admission control and adaptive coding parameters, the feedback delay, and the maximum feasible end-to-end rate that depends on channel conditions and network topology. These new results guide the system design for robust improvements of the AoI when …
In distributed machine learning (DML), the training data is distributed across multiple worker nodes to perform the underlying training in parallel. One major problem affecting the performance of DML algorithms is presence of stragglers. These are nodes that are terribly slow in performing their task which results in under-utilization of the training data that is stored in them. Towards this, gradient coding mitigates the impact of stragglers by adding sufficient redundancy in the data. Gradient coding and other straggler mitigation schemes assume that the straggler behavior of the worker node…
An adaptive randomized distributed space-time coding (DSTC) scheme is proposed for two-hop cooperative MIMO networks. Linear minimum mean square error (MMSE) receiver filters and randomized matrices subject to a power constraint are considered with an amplify-and-forward (AF) cooperation strategy. In the proposed DSTC scheme, a randomized matrix obtained by a feedback channel is employed to transform the space-time coded matrix at the relay node. The effect of the limited feedback and feedback errors are considered. Linear MMSE expressions are devised to compute the parameters of the adaptive …
Batched network coding is a low-complexity network coding solution to feedbackless multi-hop wireless packet network transmission with packet loss. The data to be transmitted is encoded into batches where each of which consists of a few coded packets. Unlike the traditional forwarding strategy, the intermediate network nodes have to perform recoding, which generates recoded packets by network coding operations restricted within the same batch. Adaptive recoding is a technique to adapt the fluctuation of packet loss by optimizing the number of recoded packets per batch to enhance the throughput…
In this paper, a joint cross-layer design of adaptive modulation and coding (AMC) and cooperative automatic repeat request (C-ARQ) scheme is proposed for a secondary user in a shared-spectrum environment. First, based on the statistical descriptions of the channel, closed-form expressions of the average spectral efficiency (SE) and the average packet loss rate (PLR) are presented. Then, the cross-layer scheme is designed, with the aim of maximizing the average SE while maintaining the average PLR under a prescribed level. An optimization problem is formed, and a sub-optimal solution is found: …
Recent technological advances in wireless communications offer new opportunities and challenges for relay network.To enhance system performance, Demodulate-Network Coding (Dm-NC) scheme has been examined at relay node; it works directly to De-map the received signals and after that forward the mixture to the destination. Simulation analysis has been proven that the performance of Dm-NC has superiority over analog-NC. In addition, the Quantize-Decode-NC scheme (QDF-NC) has been introduced. The presented simulation results clearly provide that the QDF-NC perform better than analog-NC. The toggle…
We consider the problem of distributedly computing a general class of functions, referred to as gradient-type computation, while maintaining the privacy of the input dataset. Gradient-type computation evaluates the sum of some `partial gradients', defined as polynomials of subsets of the input. It underlies many algorithms in machine learning and data analytics. We propose Harmonic Coding, which universally computes any gradient-type function, while requiring the minimum possible number of workers. Harmonic Coding strictly improves computing schemes developed based on prior works, such as Sham…
This paper presents a cross-layer approach to jointly design adaptive modulation and coding (AMC) at the physical layer and cooperative truncated automatic repeat request (ARQ) protocol at the data link layer. We first derive an exact closed form expression for the spectral efficiency of the proposed joint AMC-cooperative ARQ scheme. Aiming at maximizing this system performance measure, we then optimize an AMC scheme which directly satisfies a prescribed packet loss rate constraint at the data-link layer. The results indicate that utilizing cooperative ARQ as a retransmission strategy, noticea…
In this paper, we developed new coding strategies for the Z-channel. In particular, we look at the case with two-stage encoding. In this case, the encoder uses noiseless feedback once and adjusts the further encoding strategy based on the previous partial output of the channel. Nevertheless, the developed codes improve the known results with full feedback for small length and 1 error. A tool for the two-stage strategy is the development of a new optimality condition for non-adaptive codes.
Adaptive coding faces the following problem: given a collection of source classes such that each class in the collection has non-trivial minimax redundancy rate, can we design a single code which is asymptotically minimax over each class in the collection? In particular, adaptive coding makes sense when there is no universal code on the union of classes in the collection. In this paper, we deal with classes of sources over an infinite alphabet, that are characterized by a dominating envelope. We provide asymptotic equivalents for the redundancy of envelope classes enjoying a regular variation …
In order to meet the demands of `Internet above the clouds', we propose a multiple-antenna aided adaptive coding and modulation (ACM) for aeronautical communications. The proposed ACM scheme switches its coding and modulation mode according to the distance between the communicating aircraft, which is readily available with the aid of the airborne radar or the global positioning system. We derive an asymptotic closed-form expression of the signal-to-interference-plus-noise ratio (SINR) as the number of transmitting antennas tends to infinity, in the presence of realistic co-channel interference…
Adaptive Coding for Two-Way Wiretap Channels with One-Sided Strong Secrecy
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；International Quantum Academy (SIQA)（国际量子科学院）；Graduate School of Mathematics, Nagoya University（名古屋大学大学院数学研究科）；Volkswagen Infotainment GmbH（大众信息娱乐有限公司）
A major hurdle in machine learning is scalability to massive datasets. One approach to overcoming this is to distribute the computational tasks among several workers. \textit{Gradient coding} has been recently proposed in distributed optimization to compute the gradient of an objective function using multiple, possibly unreliable, worker nodes. By designing distributed coded schemes, gradient coded computations can be made resilient to \textit{stragglers}, nodes with longer response time comparing to other nodes in a distributed network. Most such schemes rely on operations over the real or co…
In this paper, we consider a primary and a cognitive user transmitting over a wireless fading interference channel. The primary user transmits with a constant power and utilizes an adaptive modulation and coding (AMC) scheme satisfying a bit error rate requirement. We propose a link adaptation scheme to maximize the average spectral efficiency of the cognitive radio, while a minimum required spectral efficiency for the primary user is provisioned. The resulting problem is constrained to also satisfy a bit error rate requirement and a power constraint for the cognitive link. The AMC mode select…
We propose a novel coding theoretic framework for mitigating stragglers in distributed learning. We show how carefully replicating data blocks and coding across gradients can provide tolerance to failures and stragglers for Synchronous Gradient Descent. We implement our schemes in python (using MPI) to run on Amazon EC2, and show how we compare against baseline approaches in running time and generalization error.
Although quantum key distribution (QKD) comes from the development of quantum theory, the implementation of a practical QKD system does involve a lot of classical process, such as key reconciliation and privacy amplification, which is called post-processing. Post-processing has been a crucial element to high speed QKD systems, even the bottleneck of it because of its relatively high time consumption. Low density parity check (LDPC) is now becoming a promising approach of overcoming the bottleneck due to its good performance in processing throughput. In this article we propose and simulate an e…
Gradient coding allows a master node to derive the aggregate of the partial gradients, calculated by some worker nodes over the local data sets, with minimum communication cost, and in the presence of stragglers. In this paper, for gradient coding with linear encoding, we characterize the optimum communication cost for heterogeneous distributed systems with \emph{arbitrary} data placement, with $s \in \mathbb{N}$ stragglers and $a \in \mathbb{N}$ adversarial nodes. In particular, we show that the optimum communication cost, normalized by the size of the gradient vectors, is equal to $(r-s-2a)^…
Discrete multitone modulation (DMT) systems exploit the capabilities of orthogonal subcarriers to cope efficiently with narrowband interference, high frequency attenuations and multipath fadings with the help of simple equalization filters. Adaptive linear precoded discrete multitone (LP-DMT) system is based on classical DMT, combined with a linear precoding component. In this paper, we investigate the bit and energy allocation algorithm of an adaptive LP-DMT system taking into account the channel coding scheme. A coded adaptive LPDMT system is presented in the power line communication (PLC) c…
Existing gradient coding schemes introduce identical redundancy across the coordinates of gradients and hence cannot fully utilize the computation results from partial stragglers. This motivates the introduction of diverse redundancies across the coordinates of gradients. This paper considers a distributed computation system consisting of one master and $N$ workers characterized by a general partial straggler model and focuses on solving a general large-scale machine learning problem with $L$ model parameters. We show that it is sufficient to provide at most $N$ levels of redundancies for tole…
In this study, a cross-layer design which combines adaptive modulation and coding (AMC) and hybrid automatic repeat request (HARQ) techniques for a cooperative wireless network is investigated analytically. Previous analyses of such systems in the literature are confined to the case where the fading channel is independent at each retransmission, which can be unrealistic unless the channel is varying very fast. On the other hand, temporal channel correlation can have a significant impact on the performance of HARQ systems. In this study, utilizing a Markov channel model which accounts for the t…
Gradient descent algorithms are widely used in machine learning. In order to deal with huge volume of data, we consider the implementation of gradient descent algorithms in a distributed computing setting where multiple workers compute the gradient over some partial data and the master node aggregates their results to obtain the gradient over the whole data. However, its performance can be severely affected by straggler workers. Recently, some coding-based approaches are introduced to mitigate the straggler problem, but they are efficient only when the workers are homogeneous, i.e., having the…
Gradient-based distributed learning in Parameter Server (PS) computing architectures is subject to random delays due to straggling worker nodes, as well as to possible communication bottlenecks between PS and workers. Solutions have been recently proposed to separately address these impairments based on the ideas of gradient coding, worker grouping, and adaptive worker selection. This paper provides a unified analysis of these techniques in terms of wall-clock time, communication, and computation complexity measures. Furthermore, in order to combine the benefits of gradient coding and grouping…
Group testing is the combinatorial problem of identifying the defective items in a population by grouping items into test pools. Recently, nonadaptive group testing - where all the test pools must be decided on at the start - has been studied from an information theory point of view. Using techniques from channel coding, upper and lower bounds have been given on the number of tests required to accurately recover the defective set, even when the test outcomes can be noisy. In this paper, we give the first information theoretic result on adaptive group testing - where the outcome of previous tes…
In this paper, delay-limited transmission of quasi-stationary sources over block fading channels are considered. Considering distortion outage probability as the performance measure, two source and channel coding schemes with power adaptive transmission are presented. The first one is optimized for fixed rate transmission, and hence enjoys simplicity of implementation. The second one is a high performance scheme, which also benefits from optimized rate adaptation with respect to source and channel states. In high SNR regime, the performance scaling laws in terms of outage distortion exponent a…
We present ErasureHead, a new approach for distributed gradient descent (GD) that mitigates system delays by employing approximate gradient coding. Gradient coded distributed GD uses redundancy to exactly recover the gradient at each iteration from a subset of compute nodes. ErasureHead instead uses approximate gradient codes to recover an inexact gradient at each iteration, but with higher delay tolerance. Unlike prior work on gradient coding, we provide a performance analysis that combines both delay and convergence guarantees. We establish that down to a small noise floor, ErasureHead conve…
Edge computing is emerging as a new paradigm to allow processing data at the edge of the network, where data is typically generated and collected, by exploiting multiple devices at the edge collectively. However, exploiting the potential of edge computing is challenging mainly due to the heterogeneous and time-varying nature of edge devices. Coded computation, which advocates mixing data in sub-tasks by employing erasure codes and offloading these sub-tasks to other devices for computation, is recently gaining interest, thanks to its higher reliability, smaller delay, and lower communication c…
In this paper we give a short theoretical description of the general predictive adaptive arithmetic coding technique. The links between this technique and the works of J. Rissanen in the 80's, in particular the BIC information criterion used in parametrical model selection problems, are established. We also design lossless and lossy coding techniques of images. The lossless technique uses a mix between fixed-length coding and arithmetic coding and provides better compression results than those separate methods. That technique is also seen to have an interesting application in the domain of sta…
