We investigate the upper and lower bounds on the quantization distortions for independent and identically distributed sources in the finite block-length regime. Based on the convex optimization framework of the rate-distortion theory, we derive a lower bound on the quantization distortion under finite block-length, which is shown to be greater than the asymptotic distortion given by the rate-distortion theory. We also derive two upper bounds on the quantization distortion based on random quantization codebooks, which can achieve any distortion above the asymptotic one. Moreover, we apply the new upper and lower bounds to two types of sources, the discrete binary symmetric source and the continuous Gaussian source. For the binary symmetric source, we obtain the closed-form expressions of the upper and lower bounds. For the Gaussian source, we propose a computational tractable method to numerically compute the upper and lower bounds, for both bounded and unbounded quantization codebooks.Numerical results show that the gap between the upper and lower bounds is small for reasonable block length and hence the bounds are tight.
It is proved that the limit of the normalized rate-distortion functions of block independent approximations of an irreducible, aperiodic Markoff chain is independent of the initial distribution of the Markoff chain and thus, is also equal to the rate-distortion function of the Markoff chain.
We propose a nonuniform quantized decoder for polar codes. The design metric of the quantizers is to minimize the distortion incurred by quantization. The quantizers are obtained via dynamic programming and the optimality of the quantizer is proved as well. Simulation results show that the error correction performance degradation of the proposed nonuniform quantized decoder is less than 0.1 dB compared to conventional float-point decoders under 5-bit nonuniform quantization, outperforming previous uniform quantized decoders for polar codes significantly.
We consider universal quantization with side information for Gaussian observations, where the side information is a noisy version of the sender's observation with noise variance unknown to the sender. In this paper, we propose a universally rate optimal and practical quantization scheme for all values of unknown noise variance. Our scheme uses Polar lattices from prior work, and proceeds based on a structural decomposition of the underlying auxiliaries so that even when recovery fails in a round, the parties agree on a common "reference point" that is closer than the previous one. We also pres…
In many quantization problems, the distortion function is given by the Euclidean metric to measure the distance of a source sample to any given reproduction point of the quantizer. We will in this work regard distortion functions, which are additively and multiplicatively weighted for each reproduction point resulting in a heterogeneous quantization problem, as used for example in deployment problems of sensor networks. Whereas, normally in such problems, the average distortion is minimized for given weights (parameters), we will optimize the quantization problem over all weights, i.e., we tun…
We derive a lower bound on the smallest output entropy that can be achieved via vector quantization of a $d$-dimensional source with given expected $r$th-power distortion. Specialized to the one-dimensional case, and in the limit of vanishing distortion, this lower bound converges to the output entropy achieved by a uniform quantizer, thereby recovering the result by Gish and Pierce that uniform quantizers are asymptotically optimal as the allowed distortion tends to zero. Our lower bound holds for all $d$-dimensional memoryless sources having finite differential entropy and whose integer part…
In this paper a variation of the classic vector quantization problem is considered. In the standard formulation, a quantizer is designed to minimize the distortion between input and output when the number of reconstruction points is fixed. We consider, instead, the scenario in which the number of comparators used in quantization is fixed. More precisely, we study the case in which a vector quantizer of dimension d is comprised of k comparators, each receiving a linear combination of the inputs and producing the output value one/zero if this linear combination is above/below a certain threshold…
We consider high-dimensional MIMO transmissions in frequency division duplexing (FDD) systems. For precoding, the frequency selective channel has to be measured, quantized and fed back to the base station by the users. When the number of antennas is very high this typically leads to prohibitively high quantization complexity and large feedback. In 5G New Radio (NR), a modular quantization approach has been applied for this, where first a low-dimensional subspace is identified for the whole frequency selective channel, and then subband channels are linearly mapped to this subspace and quantized…
Differential pulse-code modulation (DPCM) is recently coupled with uniform scalar quantization (SQ) to improve the rate-distortion (RD) performance for the block-based quantized compressive sensing (CS) of images. In this framework, for each block's CS measurements, a prediction is generated based on the reconstructed CS measurements of the previous blocks and subtracted from measurements of the current block in the measurement domain. The resulting residual is then quantized by uniform SQ to generate the quantization index. However, the entropy coding is still required to remove the statistic…
Asymptotic energy-distortion performance of zero-delay communication scenarios under additive white Gaussian noise is investigated. Using high-resolution analysis for quantizer design, the higher-order term in the logarithm of the distortion (termed the {\em energy-distortion dispersion}) is optimized while keeping the leading term (i.e., {\em energy-distortion exponent}) at its optimal value. For uniform and Gaussian sources, significant gains are observed compared to naïvely performed quantization, i.e., aimed at optimizing the source coding performance instead of the end-to-end distortion i…
This paper considers lossy source coding of $n$-dimensional memoryless sources and shows an explicit approximation to the minimum source coding rate required to sustain the probability of exceeding distortion $d$ no greater than $ε$, which is simpler than known dispersion-based approximations. Our approach takes inspiration in the celebrated classical result stating that the Shannon lower bound to rate-distortion function becomes tight in the limit $d \to 0$. We formulate an abstract version of the Shannon lower bound that recovers both the classical Shannon lower bound and the rate-distortion…
In this work, lossy distributed compression of pairs of correlated sources is considered. Conventionally, Shannon's random coding arguments -- using randomly generated unstructured codebooks whose blocklength is taken to be asymptotically large -- are used to derive achievability results. However, it was recently observed that in various multi-terminal communications scenarios, using random codes with constant finite blocklength may lead to improved achievable regions compared to the conventional approach. In other words, in some network communication scenarios, there is a finite optimal value…
It is known that the estimated energy consumption of digital-to analog converters (DACs) is around 30\% of the energy consumed by analog-to-digital converters (ADCs) keeping fixed the sampling rate and bit resolution. Assuming that similarly to ADC, DAC dissipation doubles with every extra bit of resolution, a decrease in two resolution bits, for instance from 4 to 2 bits, represents a 75$\% $ lower dissipation. The current limitations in sum-rates of 1-bit quantization have motivated researchers to consider extra bits in resolution to obtain higher levels of sum-rates. Following this, we devi…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Google Inc.（谷歌公司）；Allen School of Computer Science and Engineering（艾伦计算机科学与工程学院）
The enormous size of modern deep neural networks makes it challenging to deploy those models in memory and communication limited scenarios. Thus, compressing a trained model without a significant loss in performance has become an increasingly important task. Tremendous advances has been made recently, where the main technical building blocks are parameter pruning, parameter sharing (quantization), and low-rank factorization. In this paper, we propose principled approaches to improve upon the common heuristics used in those building blocks, namely pruning and quantization. We first study the fu…
Ben-Gurion University of the Negev（内盖夫本-古里安大学）；Princeton University（普林斯顿大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Weizmann Institute of Science（魏茨曼科学研究所）
Traditional deep learning models are trained at a centralized server using labeled data samples collected from end devices or users. Such data samples often include private information, which the users may not be willing to share. Federated learning (FL) is an emerging approach to train such learning models without requiring the users to share their possibly private labeled data. In FL, each user trains its copy of the learning model locally. The server then collects the individual updates and aggregates them into a global model. A major challenge that arises in this method is the need of each…
