We consider the distributed source coding problem in which correlated data picked up by scattered sensors has to be encoded separately and transmitted to a common receiver, subject to a rate-distortion constraint. Although near-tooptimal solutions based on Turbo and LDPC codes exist for this problem, in most cases the proposed techniques do not scale to networks of hundreds of sensors. We present a scalable solution based on the following key elements: (a) distortion-optimized index assignments for low-complexity distributed quantization, (b) source-optimized hierarchical clustering based on the Kullback-Leibler distance and (c) sum-product decoding on specific factor graphs exploiting the correlation of the data.
This paper considers the problem of source coding with side information at the decoder, also called Slepian-Wolf source coding scheme. In practical applications of this coding scheme, the statistical relation between the source and the side information can vary from one data transmission to another, and there is a need to adapt the coding rate depending on the current statistical relation. In this paper, we propose a novel rate-adaptive code construction based on LDPC codes for the Slepian-Wolf source coding scheme. The proposed code design method allows to optimize the code degree distributio…
Data compression has become a necessity not only the in the field of communication but also in various scientific experiments. The data that is being received is more and the processing time required has also become more. A significant change in the algorithms will help to optimize the processing speed. With the invention of Technologies like IoT and in technologies like Machine Learning there is a need to compress data. For example training an Artificial Neural Network requires a lot of data that should be processed and trained in small interval of time for which compression will be very help…
A method for encoding information in DNA sequences is described. The method is based on the precision-resolution framework, and is aimed to work in conjunction with a recently suggested terminator-free template independent DNA synthesis method. The suggested method optimizes the amount of information bits per synthesis time unit, namely, the writing rate. Additionally, the encoding scheme studied here takes into account the existence of multiple copies of the DNA sequence, which are independently distorted. Finally, quantizers for various run-length distributions are designed.
In this paper, channel optimized distributed multiple description vector quantization (CDMD) schemes are presented for distributed source coding in symmetric and asymmetric settings. The CDMD encoder is designed using a deterministic annealing approach over noisy channels with packet loss. A minimum mean squared error asymmetric CDMD decoder is proposed for effective reconstruction of a source, utilizing the side information (SI) and its corresponding received descriptions. The proposed iterative symmetric CDMD decoder jointly reconstructs the symbols of multiple correlated sources. Two types …
Sparse random linear network coding (SRLNC) is an attractive technique proposed in the literature to reduce the decoding complexity of random linear network coding. Recognizing the fact that the existing SRLNC schemes are not efficient in terms of the required reception overhead, we consider the problem of designing overhead-optimized SRLNC schemes. To this end, we introduce a new design of SRLNC scheme that enjoys very small reception overhead while maintaining the main benefit of SRLNC, i.e., its linear encoding/decoding complexity. We also provide a mathematical framework for the asymptotic…
In this paper, delay-limited transmission of quasi-stationary sources over block fading channels are considered. Considering distortion outage probability as the performance measure, two source and channel coding schemes with power adaptive transmission are presented. The first one is optimized for fixed rate transmission, and hence enjoys simplicity of implementation. The second one is a high performance scheme, which also benefits from optimized rate adaptation with respect to source and channel states. In high SNR regime, the performance scaling laws in terms of outage distortion exponent a…
This work considers optimizing LDPC codes in the physical-layer network coded two-way relay channel using noncoherent FSK modulation. The error-rate performance of channel decoding at the relay node during the multiple-access phase was improved through EXIT-based optimization of Tanner graph variable node degree distributions. Codes drawn from the DVB-S2 and WiMAX standards were used as a basis for design and performance comparison. The computational complexity characteristics of the standard codes were preserved in the optimized codes by maintaining the extended irregular repeat-accumulate (e…
College of Information and Computer Sciences, University of Massachusetts, Amherst（马萨诸塞大学阿默斯特分校信息与计算机科学学院）
Source coding is the canonical problem of data compression in information theory. In a locally encodable source coding, each compressed bit depends on only few bits of the input. In this paper, we show that a recently popular model of semi-supervised clustering is equivalent to locally encodable source coding. In this model, the task is to perform multiclass labeling of unlabeled elements. At the beginning, we can ask in parallel a set of simple queries to an oracle who provides (possibly erroneous) binary answers to the queries. The queries cannot involve more than two (or a fixed constant nu…
University of Maryland（马里兰大学）；Imperial College London（帝国理工学院）
In distributed synchronous gradient descent (GD) the main performance bottleneck for the per-iteration completion time is the slowest \textit{straggling} workers. To speed up GD iterations in the presence of stragglers, coded distributed computation techniques are implemented by assigning redundant computations to workers. In this paper, we propose a novel gradient coding (GC) scheme that utilizes dynamic clustering, denoted by GC-DC, to speed up the gradient calculation. Under time-correlated straggling behavior, GC-DC aims at regulating the number of straggling workers in each cluster based …
Qualcomm AI Research（高通人工智能研究院）；Qualcomm Technologies, Inc.（高通技术公司）
Neural-based image and video codecs are significantly more power-efficient when weights and activations are quantized to low-precision integers. While there are general-purpose techniques for reducing quantization effects, large losses can occur when specific entropy coding properties are not considered. This work analyzes how entropy coding is affected by parameter quantizations, and provides a method to minimize losses. It is shown that, by using a certain type of coding parameters to be learned, uniform quantization becomes practically optimal, also simplifying the minimization of code memo…
In distributed storage systems reliability is achieved through redundancy stored at different nodes in the network. Then a data collector can reconstruct source information even though some nodes fail. To maintain reliability, an autonomous and efficient protocol should be used to repair the failed node. The repair process causes traffic and consequently transmission cost in the network. Recent results found the optimal trafficstorage tradeoff, and proposed regenerating codes to achieve the optimality. We aim at minimizing the transmission cost in the repair process. We consider the network to…
In this paper, we introduce further our recently designed family of L2 orthogonal Space-Time codes for CPM. With their advantage of maintaining both the constant envelope properties of CPM, the diversity of Space-Time codes and moreover orthogonality, and thus reduced decoding complexity, these codes are also full rate, even for more than two transmitting antennas. The issue of power efficiency for these codes is first dealt with by proving that the inherent increase in bandwidth in these systems is quite moderate. It is then detailed how the initial state of the code influences the coding gai…
Imperial College London（帝国理工学院）；University of Maryland, College Park（马里兰大学帕克分校）
Gradient descent (GD) methods are commonly employed in machine learning problems to optimize the parameters of the model in an iterative fashion. For problems with massive datasets, computations are distributed to many parallel computing servers (i.e., workers) to speed up GD iterations. While distributed computing can increase the computation speed significantly, the per-iteration completion time is limited by the slowest straggling workers. Coded distributed computing can mitigate straggling workers by introducing redundant computations; however, existing coded computing schemes are mainly d…
A status updating system is considered in which a variable length code is used to transmit messages to a receiver over a noisy channel. The goal is to optimize the codewords lengths such that successfully-decoded messages are timely. That is, such that the age-of-information (AoI) at the receiver is minimized. A hybrid ARQ (HARQ) scheme is employed, in which variable-length incremental redundancy (IR) bits are added to the originally-transmitted codeword until decoding is successful. With each decoding attempt, a non-zero processing delay is incurred. The optimal codewords lengths are analytic…
We are interested in how to best communicate a real valued source to a number of destinations (sinks) over a network with capacity constraints in a collective fidelity metric over all the sinks, a problem which we call joint network-source coding. It is demonstrated that multiple description codes along with proper diversity routing provide a powerful solution to joint network-source coding. A systematic optimization approach is proposed. It consists of optimizing the network routing given a multiple description code and designing optimal multiple description code for the corresponding optimiz…
Polar codes are considered the latest major breakthrough in coding theory. Polar codes were introduced by Arıkan in 2008. In this letter, we show that the binary polar codes are the same as the optimized codes for bitwise multistage decoding (OCBM), which have been discovered before by Stolte in 2002. The equivalence between the techniques used for the constructions and decodings of both codes is established.
In this paper, we present iterative algorithms that numerically compute the rate-distortion regions of two problems: the two-encoder multiterminal source coding problem and the Chief Executive Officer (CEO) problem, both under logarithmic loss distortion measure.With the clear connection of these models with the distributed information bottleneck method, the proposed algorithms may find usefulness in a variety of applications, such as clustering, pattern recognition and learning. We illustrate the efficiency of our algorithms through some numerical examples.
University of Maryland（马里兰大学）；Imperial College London（伦敦帝国学院）
Distributed implementations are crucial in speeding up large scale machine learning applications. Distributed gradient descent (GD) is widely employed to parallelize the learning task by distributing the dataset across multiple workers. A significant performance bottleneck for the per-iteration completion time in distributed synchronous GD is $straggling$ workers. Coded distributed computation techniques have been introduced recently to mitigate stragglers and to speed up GD iterations by assigning redundant computations to workers. In this paper, we consider gradient coding (GC), and propose …
This paper investigates delay-distortion-power trade offs in transmission of quasi-stationary sources over block fading channels by studying encoder and decoder buffering techniques to smooth out the source and channel variations. Four source and channel coding schemes that consider buffer and power constraints are presented to minimize the reconstructed source distortion. The first one is a high performance scheme, which benefits from optimized source and channel rate adaptation. In the second scheme, the channel coding rate is fixed and optimized along with transmission power with respect to…
We consider the energy savings that can be obtained by employing network coding instead of plain routing in wireless multiple unicast problems. We establish lower bounds on the benefit of network coding, defined as the maximum of the ratio of the minimum energy required by routing and network coding solutions, where the maximum is over all configurations. It is shown that if coding and routing solutions are using the same transmission range, the benefit in $d$-dimensional networks is at least $2d/\lfloor\sqrt{d}\rfloor$. Moreover, it is shown that if the transmission range can be optimized for…
Many applications require data processing to be performed on individual pieces of data which are of finite sizes, e.g., files in cloud storage units and packets in data networks. However, traditional universal compression solutions would not perform well over the finite-length sequences. Recently, we proposed a framework called memory-assisted universal compression that holds a significant promise for reducing the amount of redundant data from the finite-length sequences. The proposed compression scheme is based on the observation that it is possible to learn source statistics (by memorizing p…
We introduce the Fixed Cluster Repair System (FCRS) as a novel architecture for Distributed Storage Systems (DSS), achieving a small repair bandwidth while guaranteeing a high availability. Specifically we partition the set of servers in a DSS into $s$ clusters and allow a failed server to choose any cluster other than its own as its repair group. Thereby, we guarantee an availability of $s-1$. We characterize the repair bandwidth vs. storage trade-off for the FCRS under functional repair and show that the minimum repair bandwidth can be improved by an asymptotic multiplicative factor of $2/3$…
In bidirectional relaying using Physical Layer Network Coding (PLNC), it is generally assumed that users employ same modulation schemes in the Multiple Access phase. However, as observed by Zhang et al., it may not be desirable for the users to always use the same modulation schemes, particularly when user-relay channels are not equally strong. Such a scheme is called Heterogeneous PLNC. However, the approach in [1] uses the computationally intensive Closest Neighbour Clustering (CNC) algorithm to find the network coding maps to be applied at the relay. Also, the treatment is specific to certa…
We develop the theory and practical implementation of p-adic sparse coding of data. Rather than the standard, sparsifying criterion that uses the $L_0$ pseudo-norm, we use the p-adic norm. We require that the hierarchy or tree be node-ranked, as is standard practice in agglomerative and other hierarchical clustering, but not necessarily with decision trees. In order to structure the data, all computational processing operations are direct reading of the data, or are bounded by a constant number of direct readings of the data, implying linear computational time. Through p-adic sparse data codin…
Zero-delay transmission of a Gaussian source over an additive white Gaussian noise (AWGN) channel is considered in the presence of an additive Gaussian interference signal. The mean squared error (MSE) distortion is minimized under an average power constraint assuming that the interference signal is known at the transmitter. Optimality of simple linear transmission does not hold in this setting due to the presence of the known interference signal. While the optimal encoder-decoder pair remains an open problem, various non-linear transmission schemes are proposed in this paper. In particular, i…
There are two fundamentally different fronthaul techniques in the downlink communication of cloud radio access network (C-RAN): the data-sharing strategy and the compression-based strategy. Under the former strategy, each user's message is multicast from the central processor (CP) to all the serving remote radio heads (RRHs) over the fronthaul network, which then cooperatively serve the users through joint beamforming; while under the latter strategy, the user messages are first beamformed then quantized at the CP, and the compressed signal is unicast to the corresponding RRH, which then decom…
An important ingredient of the future 5G systems will be Ultra-Reliable Low-Latency Communication (URLLC). A way to offer URLLC without intervention in the baseband/PHY layer design is to use \emph{interface diversity} and integrate multiple communication interfaces, each interface based on a different technology. Our approach is to use coding to seamlessly distribute coded payload and redundancy data across multiple available communication interfaces. We formulate an optimization problem to find the payload allocation weights that maximize the reliability at specific target latency values. By…
A noncoherent two-way relaying system is developed using physical-layer network coding for improved throughput over conventional relaying in a fading channel. Energy-efficient noncoherent operation is achieved using multitone frequency shift keying (FSK). A novel soft-output demodulator is developed for the relay, and corresponding achievable exchange rates are found for Rayleigh fading and AWGN channels. Bit-error rate performance approaching the achievable rate is realized using a capacity-approaching channel code and a receiver architecture that iterates between demodulation and channel dec…
Intra-session network coding has been shown to offer significant gains in terms of achievable throughput and delay in settings where one source multicasts data to several clients. In this paper, we consider a more general scenario where multiple sources transmit data to sets of clients and study the benefits of inter-session network coding, when network nodes have the opportunity to combine packets from different sources. In particular, we propose a novel framework for optimal rate allocation in inter-session network coding systems. We formulate the problem as the minimization of the average d…
Beijing University of Posts and Telecommunications（北京邮电大学）；Peng Cheng Laboratory（鹏城实验室）
As one novel approach to realize end-to-end wireless image semantic transmission, deep learning-based joint source-channel coding (deep JSCC) method is emerging in both deep learning and communication communities. However, current deep JSCC image transmission systems are typically optimized for traditional distortion metrics such as peak signal-to-noise ratio (PSNR) or multi-scale structural similarity (MS-SSIM). But for low transmission rates, due to the imperfect wireless channel, these distortion metrics lose significance as they favor pixel-wise preservation. To account for human visual pe…
Inspired by recent work on convex formulations of clustering (Lashkari & Golland, 2008; Nowozin & Bakir, 2008) we investigate a new formulation of the Sparse Coding Problem (Olshausen & Field, 1997). In sparse coding we attempt to simultaneously represent a sequence of data-vectors sparsely (i.e. sparse approximation (Tropp et al., 2006)) in terms of a 'code' defined by a set of basis elements, while also finding a code that enables such an approximation. As existing alternating optimization procedures for sparse coding are theoretically prone to severe local minima problems, we propose a conv…
We wish to minimize the resources used for network coding while achieving the desired throughput in a multicast scenario. We employ evolutionary approaches, based on a genetic algorithm, that avoid the computational complexity that makes the problem NP-hard. Our experiments show great improvements over the sub-optimal solutions of prior methods. Our new algorithms improve over our previously proposed algorithm in three ways. First, whereas the previous algorithm can be applied only to acyclic networks, our new method works also with networks with cycles. Second, we enrich the set of components…
City University of Hong Kong（香港城市大学）；Japan Institute of Science and Technology (JAIST)（日本北陆先端科学技术大学院大学）；Huawei Technologies Co., Ltd.（华为技术有限公司）
This paper investigates a large unitarily invariant system (LUIS) involving a unitarily invariant sensing matrix, an arbitrary fixed signal distribution, and forward error control (FEC) coding. Several area properties are established based on the state evolution of orthogonal approximate message passing (OAMP) in an un-coded LUIS. Under the assumptions that the state evolution for joint OAMP and FEC decoding is correct and the replica method is reliable, we analyze the achievable rate of OAMP. We prove that OAMP reaches the constrained capacity predicted by the replica method of the LUIS with …
This paper considers uplink multiple access (MA) transmissions, where the MA technique is adaptively selected between Non Orthogonal Multiple Access (NOMA) and Orthogonal Multiple Access (OMA). Two types of users, namely Internet of Things (IoT) and enhanced mobile broadband (eMBB) coexist with different metrics to be optimized, energy efficiency (EE) for IoT and spectral efficiency (SE) for eMBB. The corresponding multi-objective power allocation problems aiming at maximizing a weighted sum of EE and SE are solved for both NOMA and OMA. Based on the identification of the best MA strategy, a c…
University of California, Berkeley（加州大学伯克利分校）；Tsinghua University（清华大学）
To learn intrinsic low-dimensional structures from high-dimensional data that most discriminate between classes, we propose the principle of Maximal Coding Rate Reduction ($\text{MCR}^2$), an information-theoretic measure that maximizes the coding rate difference between the whole dataset and the sum of each individual class. We clarify its relationships with most existing frameworks such as cross-entropy, information bottleneck, information gain, contractive and contrastive learning, and provide theoretical guarantees for learning diverse and discriminative features. The coding rate can be ac…
