A major hurdle in machine learning is scalability to massive datasets. One approach to overcoming this is to distribute the computational tasks among several workers. \textit{Gradient coding} has been recently proposed in distributed optimization to compute the gradient of an objective function using multiple, possibly unreliable, worker nodes. By designing distributed coded schemes, gradient coded computations can be made resilient to \textit{stragglers}, nodes with longer response time comparing to other nodes in a distributed network. Most such schemes rely on operations over the real or complex numbers and are inherently numerically unstable. We present a binary scheme which avoids such operations, thereby enabling numerically stable distributed computation of the gradient. Also, some restricting assumptions in prior work are dropped, and a more efficient decoding is given.
This work presents a polar coding scheme for fading channels, focusing primarily on fading binary symmetric and additive exponential noise channels. For fading binary symmetric channels, a hierarchical coding scheme is presented, utilizing polar coding both over channel uses and over fading blocks. The receiver uses its channel state information (CSI) to distinguish states, thus constructing an overlay erasure channel over the underlying fading channels. By using this scheme, the capacity of a fading binary symmetric channel is achieved without CSI at the transmitter. Noting that a fading AWGN…
An $l$-link binary CEO problem is considered in this paper. We present a practical encoding and decoding scheme for this problem employing the graph-based codes. A successive coding scheme is proposed for converting an $l$-link binary CEO problem to the $(2l-1)$ single binary Wyner-Ziv (WZ) problems. By using the compound LDGM-LDPC codes, the theoretical bound of each binary WZ is asymptotically achievable. Our proposed decoder successively decodes the received data by employing the well-known Sum-Product (SP) algorithm and leverages them to reconstruct the source. The sum-rate distortion perf…
The paper provides a new perspective on peak- and average-constrained Gaussian channels. Such channels model optical wireless communication (OWC) systems which employ intensity-modulation with direct detection (IM/DD). First, the paper proposes a new, capacity-preserving vector binary channel (VBC) model, consisting of dependent binary noisy bit-pipes. Then, to simplify coding over this VBC, the paper proposes coding schemes with varying levels of complexity, building on the capacity of binary-symmetric channels (BSC) and channels with state. The achievable rates are compared to capacity and c…
Binary LDPC coded relay systems have been well studied previously with the assumption of infinite codeword length. In this paper, we deal with non-binary LDPC codes which can outperform their binary counterpart especially for practical codeword length. We utilize non-binary LDPC codes and recently invented non-binary coding techniques known as multiplicative repetition to design the low-rate coding strategy for the decode-and-forward half-duplex relay channel. We claim that the proposed strategy is simple since the destination and the relay can decode with almost the same computational complex…
We propose a new method to embed data in binary images, including scanned text, figures, and signatures. Our method relies on the concept of wet paper codes. The shuffling before embedding is used in order to equalize irregular embedding capacity from diverse areas in the image. The hidden data can be extracted without the original binary image. We illustrate some examples of watermarked binary images after wet paper coding.
In this paper, a multi-source multi-relay cooperative wireless network with binary modulation and binary network coding is studied. The system model encompasses: i) a demodulate-and-forward protocol at the relays, where the received packets are forwarded regardless of their reliability; and ii) a maximum-likelihood optimum demodulator at the destination, which accounts for possible demodulations errors at the relays. An asymptotically-tight and closed-form expression of the end-to-end error probability is derived, which clearly showcases diversity order and coding gain of each source. Unlike o…
We derive the capacity of the binary multi-way relay channel, in which multiple users exchange messages at a common rate through a relay. The capacity is achieved using a novel functional-decode-forward coding strategy. In the functional-decode-forward coding strategy, the relay decodes functions of the users' messages without needing to decode individual messages. The functions to be decoded by the relay are defined such that when the relay broadcasts the functions back to the users, every user is able to decode the messages of all other users.
We address the problem of constructing of coding schemes for the channels with high-order modulations. It is known, that non-binary LDPC codes are especially good for such channels and significantly outperform their binary counterparts. Unfortunately, their decoding complexity is still large. In order to reduce the decoding complexity we consider multilevel coding schemes based on non-binary LDPC codes (NB-LDPC-MLC schemes) over smaller fields. The use of such schemes gives us a reasonable gain in complexity. At the same time the performance of NB-LDPC-MLC schemes is practically the same as th…
A pruned variant of polar coding is reinvented for all binary erasure channels. For small $\varepsilon>0$, we construct codes with block length $\varepsilon^{-5}$, code rate $\text{Capacity}-\varepsilon$, error probability $\varepsilon$, and encoding and decoding time complexity $O(N\log|\log\varepsilon|)$ per block, equivalently $O(\log|\log\varepsilon|)$ per information bit (Propositions 5 to 8). This result also follows if one applies systematic polar coding [Arıkan 10.1109/LCOMM.2011.061611.110862] with simplified successive cancelation decoding [Alamdar-Yazdi-Kschischang 10.1109/LCOMM.201…
In this paper, we first propose an universal polar coding scheme for parallel Gaussian channels with non-binary inputs. It is assumed that the encoder knows only the sum capacity of M parallel channels instead of the capacity of any single channel. By decomposing each parallel channel into T = [log2r] sub channels, we therefore obtain MT binary sub-channels. A super polar coding scheme that across all sub-channels is then proposed. This scheme can achieve the sum capacity when the block length is sufficiently large. We have also discussed the applications of parallel polar coding design for bo…
The problem of communication over binary dirty paper (DP) using nested polar codes is considered. An improved scheme, focusing on low delay, short to moderate blocklength communication is proposed. Successive cancellation list (SCL) decoding with properly defined CRC is used for channel coding, and SCL encoding without CRC is used for source coding. The performance is compared to the best achievable rate of any coding scheme for binary DP using nested codes. A well known problem with nested polar codes for binary DP is the existence of frozen channel code bits that are not frozen in the source…
The $L$-link binary Chief Executive Officer (CEO) problem under logarithmic loss is investigated in this paper. A quantization splitting technique is applied to convert the problem under consideration to a $(2L-1)$-step successive Wyner-Ziv (WZ) problem, for which a practical coding scheme is proposed. In the proposed scheme, low-density generator-matrix (LDGM) codes are used for binary quantization while low-density parity-check (LDPC) codes are used for syndrome generation; the decoder performs successive decoding based on the received syndromes and produces a soft reconstruction of the remo…
We give an algorithm for finding network encoding and decoding equations for error-free multicasting networks with multiple sources and sinks. The algorithm given is efficient (polynomial complexity) and works on any kind of network (acyclic, link cyclic, flow cyclic, or even in the presence of knots). The key idea will be the appropriate use of the delay (both natural and additional) during the encoding. The resulting code will always work with finite delay with binary encoding coefficients.
A polar coding scheme is introduced in this paper for the wire-tap channel. It is shown that the provided scheme achieves the entire rate-equivocation region for the case of symmetric and degraded wire-tap channel, where the weak notion of secrecy is assumed. For the particular case of binary erasure wire-tap channel, an alternative proof is given. The case of general non-degraded wire-tap channels is also considered.
A polar coding scheme for fading channels is proposed in this paper. More specifically, the focus is Gaussian fading channel with a BPSK modulation technique, where the equivalent channel could be modeled as a binary symmetric channel with varying cross-over probabilities. To deal with variable channel states, a coding scheme of hierarchically utilizing polar codes is proposed. In particular, by observing the polarization of different binary symmetric channels over different fading blocks, each channel use corresponding to a different polarization is modeled as a binary erasure channel such th…
We consider lossy source compression of a binary symmetric source using polar codes and the low-complexity successive encoding algorithm. It was recently shown by Arikan that polar codes achieve the capacity of arbitrary symmetric binary-input discrete memoryless channels under a successive decoding strategy. We show the equivalent result for lossy source compression, i.e., we show that this combination achieves the rate-distortion bound for a binary symmetric source. We further show the optimality of polar codes for various problems including the binary Wyner-Ziv and the binary Gelfand-Pinske…
We consider polar codes for memoryless sources with side information and show that the blocklength, construction, encoding and decoding complexities are bounded by a polynomial of the reciprocal of the gap between the compression rate and the conditional entropy. This extends the recent results of Guruswami and Xia to a slightly more general setting, which in turn can be applied to (1) sources with non-binary alphabets, (2) key generation for discrete and Gaussian sources, and (3) Slepian-Wolf coding and multiple accessing. In each of these cases, the complexity scaling with respect to the num…
This paper considers the memoryless input-constrained binary erasure channel (BEC). The channel input constraint is the $(d,\infty)$-runlength limited (RLL) constraint, which mandates that any pair of successive $1$s in the input sequence be separated by at least $d$ $0$s. We consider a scenario where there is causal, noiseless feedback from the decoder. We demonstrate a simple, labelling-based, zero-error feedback coding scheme, which we prove to be feedback capacity-achieving, and, as a by-product, obtain an explicit characterization of the feedback capacity. Our proof is based on showing th…
We devise an analytically simple as well as invertible approximate expression, which describes the relation between the minimum distance of a binary code and the corresponding maximum attainable code-rate. For example, for a rate-(1/4), length-256 binary code the best known bounds limit the attainable minimum distance to 65<d(n=256,k=64)<90, while our solution yields d(n=256,k=64)=74.4. The proposed formula attains the approximation accuracy within the rounding error for ~97% of (n,k) scenarios, where the exact value of the minimum distance is known. The results provided may be utilized for th…
We consider binary systematic network codes and investigate their capability of decoding a source message either in full or in part. We carry out a probability analysis, derive closed-form expressions for the decoding probability and show that systematic network coding outperforms conventional network coding. We also develop an algorithm based on Gaussian elimination that allows progressive decoding of source packets. Simulation results show that the proposed decoding algorithm can achieve the theoretical optimal performance. Furthermore, we demonstrate that systematic network codes equipped w…
In steganography, we always hope to maximize the embedding payload subject to an upper-bounded distortion. We need suitable distortion measurement to evaluate the embedding impact. However, different distortion functions exposes different levels of distortion evaluation, implying that we have different optimization distributions by applying different distortion functions. In applications, the embedding distortion is caused by a certain number of embedding operations. Instead of a predefined distortion, we actually utilize a number of modifications to embed as many message bits as possible as l…
In this paper we study binary trees with choosable edge lengths, in particular rooted binary trees with the property that the two edges leading from every non-leaf to its two children are assigned integral lengths $l_1$ and $l_2$ with $l_1+l_2 =k$ for a constant $k\in\mathbb{N}$. The depth of a leaf is the total length of the edges of the unique root-leaf-path. We present a generalization of the Huffman Coding that can decide in polynomial time if for given values $d_1,...,d_n\in\mathbb{N}_{\geq 0}$ there exists a rooted binary tree with choosable edge lengths with $n$ leaves having depths at …
We consider multi--hop networks comprising Binary Symmetric Channels ($\mathsf{BSC}$s). The network carries unicast flows for multiple users. The utility of the network is the sum of the utilities of the flows, where the utility of each flow is a concave function of its throughput. Given that the network capacity is shared by the flows, there is a contention for network resources like coding rate (at the physical layer), scheduling time (at the MAC layer), etc., among the flows. We propose a proportional fair transmission scheme that maximises the sum utility of flow throughputs subject to the…
We study the application of polar codes in deletion channels by analyzing the cascade of a binary erasure channel (BEC) and a deletion channel. We show how polar codes can be used effectively on a BEC with a single deletion, and propose a list decoding algorithm with a cyclic redundancy check for this case. The decoding complexity is $O(N^2\log N)$, where $N$ is the blocklength of the code. An important contribution is an optimization of the amount of redundancy added to minimize the overall error probability. Our theoretical results are corroborated by numerical simulations which show that th…
We provide a practical implementation of the rubber method of Ahlswede et al. for binary alphabets. The idea is to create the "skeleton" sequence therein via an arithmetic decoder designed for a particular $k$-th order Markov chain. For the stochastic binary symmetric channel, we show that the scheme is nearly optimal in a strong sense for certain parameters.
Given a noiseless protocol $π_0$ computing a function $f(x, y)$ of Alice and Bob's private inputs $x, y$, the goal of interactive coding is to construct an error-resilient protocol $π$ computing $f$ such that even if some fraction of the communication is adversarially corrupted, both parties still learn $f(x, y)$. Ideally, the resulting scheme $π$ should be positive rate, computationally efficient, and achieve optimal error resilience. While interactive coding over large alphabets is well understood, the situation over the binary alphabet has remained evasive. At the present moment, the known …
Motivated by applications in DNA-based storage, we study explicit encoding and decoding schemes of binary strings satisfying locally balanced constraints, where the $(\ell,δ)$-locally balanced constraint requires that the weight of any consecutive substring of length $\ell$ is between $\frac{\ell}{2}-δ$ and $\frac{\ell}{2}+δ$. In this paper we present coding schemes for the strongly locally balanced constraints and the locally balanced constraints, respectively. Moreover, we introduce an additional result on the linear recurrence formula of the number of binary strings which are $(6,1)$-locall…
This paper addresses the problem of Approximate Nearest Neighbor (ANN) search in pattern recognition where feature vectors in a database are encoded as compact codes in order to speed-up the similarity search in large-scale databases. Considering the ANN problem from an information-theoretic perspective, we interpret it as an encoding, which maps the original feature vectors to a less entropic sparse representation while requiring them to be as informative as possible. We then define the coding gain for ANN search using information-theoretic measures. We next show that the classical approach t…
In this paper, we contribute to the theoretical understanding, the design, and the performance evaluation of multi-source multi-relay network-coded cooperative diversity protocols. These protocols are useful to counteract the spectral inefficiency of repetition-based cooperation. We provide a general analytical framework for analysis and design of wireless networks using the Demodulate-and-Forward (DemF) protocol with binary Network Coding (NC) at the relays and Cooperative Maximal Ratio Combining (C-MRC) at the destination. Our system model encompasses an arbitrary number of relays which offe…
This paper studies coding schemes for the $q$-ary symmetric channel based on binary low-density parity-check (LDPC) codes that work for any alphabet size $q=2^m$, $m\in\mathbb{N}$, thus complementing some recently proposed packet-based schemes requiring large $q$. First, theoretical optimality of a simple layered scheme is shown, then a practical coding scheme based on a simple modification of standard binary LDPC decoding is proposed. The decoder is derived from first principles and using a factor-graph representation of a front-end that maps $q$-ary symbols to groups of $m$ bits connected to…
In this paper the multicasting of independent parallel Gaussian sources over a binary erasure broadcasted channel is considered. Multiresolution embedded quantizer and layered joint source-channel coding schemes are used in order to serve simultaneously several users at different channel capacities. The convex nature of the rate-distortion function, computed by means of reverse water-filling, allows us to solve relevant convex optimization problems corresponding to different performance criteria. Then, layered joint source-channel codes are constructed based on the concatenation of embedded sc…
This work gives an explicit construction of a family of error correcting codes for the binary deletion channel and for the Poisson repeat channel. In the binary deletion channel with parameter $p$ (BDC$_p$) every bit is deleted independently with probability $p$. A lower bound of $(1-p)/9$ is known on the capacity of the BDC$_p$ \cite{mitzenmacher2006simple}, yet no explicit construction is known to achieve this rate. We give an explicit family of codes of rate $(1-p)/16$, for every $p$. This improves upon the work of Guruswami and Li \cite{guruswami2017efficiently} that gave a construction of…
This paper describes a new method of data encoding which may be used in various modern digital, computer and telecommunication systems and devices. The method permits the compression of data for storage or transmission, allowing the exact original data to be reconstructed without any loss of content. The method is characterized by the simplicity of implementation, as well as high speed and compression ratio. The method is based on a unique scheme of binary-ternary prefix-free encoding of characters of the original data. This scheme does not require the transmission of the code tables from enco…
We present nonasymptotic achievability and converse bounds on the maximum coding rate (for a fixed average error probability and a fixed average blocklength) of variable-length full-feedback (VLF) and variable-length stop-feedback (VLSF) codes operating over a binary erasure channel (BEC). For the VLF setup, the achievability bound relies on a scheme that maps each message onto a variable-length Huffman codeword and then repeats each bit of the codeword until it is received correctly. The converse bound is inspired by the meta-converse framework by Polyanskiy, Poor, and Verdú (2010) and relies…
In this paper, we propose an approach to numerically compute the feedback capacity of stationary finite dimensional Gaussian channels and construct (arbitrarily close to) capacity-achieving feedback codes. In particular, we first extend the interpretation of feedback communication over stationary finite dimensional Gaussian channels as feedback control systems by showing that, the problem of finding stabilizing feedback controllers with maximal reliable transmission rate over Youla parameters coincides with the problem of finding strictly causal filters to achieve feedback capacity derived in …
The interest in channel models in which the data is sent as an unordered set of binary strings has increased lately, due to emerging applications in DNA storage, among others. In this paper we analyze the minimal redundancy of binary codes for this channel under substitution errors, and provide several constructions, some of which are shown to be asymptotically optimal up to constants. The surprising result in this paper is that while the information vector is sliced into a set of unordered strings, the amount of redundant bits that are required to correct errors is order-wise equivalent to th…
We develop a low-complexity coding scheme to achieve covert communications over binary-input discrete memoryless channels (BI-DMCs). We circumvent the impossibility of covert communication with linear codes by introducing non-linearity through the use of pulse position modulation (PPM) and multilevel coding (MLC). We show that the MLC-PPM scheme exhibits many appealing properties; in particular, the channel at a given index level remains stationary as the number of level increases, which allows one to use families of channel capacity- and channel resolvability-achieving codes to concretely ins…
We present DeepCABAC, a novel context-adaptive binary arithmetic coder for compressing deep neural networks. It quantizes each weight parameter by minimizing a weighted rate-distortion function, which implicitly takes the impact of quantization on to the accuracy of the network into account. Subsequently, it compresses the quantized values into a bitstream representation with minimal redundancies. We show that DeepCABAC is able to reach very high compression ratios across a wide set of different network architectures and datasets. For instance, we are able to compress by x63.6 the VGG16 ImageN…
We introduce a discrete network corresponding to any Gaussian wireless network that is obtained by simply quantizing the received signals and restricting the transmitted signals to a finite precision. Since signals in the discrete network are obtained from those of a Gaussian network, the Gaussian network can be operated on the quantization-based digital interface defined by the discrete network. We prove that this digital interface is near-optimal for Gaussian relay networks and the capacities of the Gaussian and the discrete networks are within a bounded gap of O(M^2) bits, where M is the nu…
In this paper, we consider the problem of polar coding for block fading channels, with emphasis on those with instantaneous channel state information (CSI) at neither the transmitter nor the receiver. Our approach is to decompose a block fading channel of $T_c$ symbols per coherent interval into $T_c$ binary-input sub-channels in a capacity-preserving way, and design a polar code for each of them. For the case when instantaneous CSI is available at the receiver, a random interleaver can be used to enable joint encoding and decoding of all sub-channels so as to enhance the finite length perform…
In this paper, we propose an efficient coding scheme for the two-link binary Chief Executive Officer (CEO) problem under logarithmic loss criterion. The exact rate-distortion bound for a two-link binary CEO problem under the logarithmic loss has been obtained by Courtade and Weissman. We propose an encoding scheme based on compound LDGM-LDPC codes to achieve the theoretical bounds. In the proposed encoding, a binary quantizer using LDGM codes and a syndrome-coding employing LDPC codes are applied. An iterative joint decoding is also designed as a fusion center. The proposed CEO decoder is base…
Advancing over up-to-date information theoretic results that assume symmetric correlation models, in this work we consider the problem of lossy binary source coding with side information, where the correlation is expressed by a generic binary asymmetric channel. Specifically, we present an in-depth analysis of rate distortion with side information available to both the encoder and decoder (conventional predictive), as well as the Wyner-Ziv problem for this particular setup. Prompted by our recent results for the Z-channel correlation case, we evaluate the rate loss between the Wyner-Ziv and th…
We investigate the privacy of two approaches to (biometric) template protection: Helper Data Systems and Sparse Ternary Coding with Ambiguization. In particular, we focus on a privacy property that is often overlooked, namely how much leakage exists about one specific binary property of one component of the feature vector. This property is e.g. the sign or an indicator that a threshold is exceeded. We provide evidence that both approaches are able to protect such sensitive binary variables, and discuss how system parameters need to be set.
In this paper, a compressive sensing (CS) approach is proposed for sparse binary signals' compression and reconstruction based on analog fountain codes (AFCs). In the proposed scheme, referred to as the analog fountain compressive sensing (AFCS), each measurement is generated from a linear combination of L randomly selected binary signal elements with real weight coefficients. The weight coefficients are chosen from a finite weight set and L, called measurement degree, is obtained based on a predefined degree distribution function. We propose a simple verification based reconstruction algorith…
Boolean functions have important applications in cryptography and coding theory. Two famous classes of binary codes derived from Boolean functions are the Reed-Muller codes and Kerdock codes. In the past two decades, a lot of progress on the study of applications of Boolean functions in coding theory has been made. Two generic constructions of binary linear codes with Boolean functions have been well investigated in the literature. The objective of this paper is twofold. The first is to provide a survey on recent results, and the other is to propose open problems on one of the two generic cons…
In this paper, we present iterative algorithms that numerically compute the rate-distortion regions of two problems: the two-encoder multiterminal source coding problem and the Chief Executive Officer (CEO) problem, both under logarithmic loss distortion measure.With the clear connection of these models with the distributed information bottleneck method, the proposed algorithms may find usefulness in a variety of applications, such as clustering, pattern recognition and learning. We illustrate the efficiency of our algorithms through some numerical examples.
We consider oblivious transfer protocols performed over binary symmetric channels in a malicious setting where parties will actively cheat if they can. We provide constructions purely based on coding theory that achieve an explicit positive rate, the essential ingredient being the existence of linear codes whose Schur products are asymptotically good.
