This paper studies the tradeoff in privacy and utility in a single-trial multi-terminal guessing (estimation) framework using a system model that is inspired by index coding. There are $n$ independent discrete sources at a data curator. There are $m$ legitimate users and one adversary, each with some side information about the sources. The data curator broadcasts a distorted function of sources to legitimate users, which is also overheard by the adversary. In terms of utility, each legitimate user wishes to perfectly reconstruct some of the unknown sources and attain a certain gain in the estimation correctness for the remaining unknown sources. In terms of privacy, the data curator wishes to minimize the maximal leakage: the worst-case guessing gain of the adversary in estimating any target function of its unknown sources after receiving the broadcast data. Given the system settings, we derive fundamental performance lower bounds on the maximal leakage to the adversary, which are inspired by the notion of confusion graph and performance bounds for the index coding problem. We also detail a greedy privacy enhancing mechanism, which is inspired by the agglomerative clustering algorithms in the information bottleneck and privacy funnel problems.
A novel definition of the conditional smooth Renyi entropy, which is different from that of Renner and Wolf, is introduced. It is shown that our definition of the conditional smooth Renyi entropy is appropriate to give lower and upper bounds on the optimal guessing moment in a guessing problem where the guesser is allowed to stop guessing and declare an error. Further a general formula for the optimal guessing exponent is given. In particular, a single-letterized formula for mixture of i.i.d. sources is obtained. Another application in the problem of source coding with the common side-informat…
Three science and engineering problems of recent interests -index coding, locally recoverable distributed storage, and guessing games on graphs- are discussed and the connection between their optimal solutions is elucidated. By generalizing recent results by Shanmugam and Dimakis and by Mazumdar on the complementarity between the optimal broadcast rate of an index coding problem on a directed graph and the normalized rate of a locally recoverable distributed storage problem on the same graph, it is shown that the capacity region and the optimal rate region of these two problems are complementa…
This paper provides upper and lower bounds on the optimal guessing moments of a random variable taking values on a finite set when side information may be available. These moments quantify the number of guesses required for correctly identifying the unknown object and, similarly to Arikan's bounds, they are expressed in terms of the Arimoto-Rényi conditional entropy. Although Arikan's bounds are asymptotically tight, the improvement of the bounds in this paper is significant in the non-asymptotic regime. Relationships between moments of the optimal guessing function and the MAP error probabili…
This paper establishes a close relationship among the four information theoretic problems, namely Campbell source coding, Arikan guessing, Huleihel et al. memoryless guessing and Bunte and Lapidoth tasks partitioning problems. We first show that the aforementioned problems are mathematically related via a general moment minimization problem whose optimum solution is given in terms of Renyi entropy. We then propose a general framework for the mismatched version of these problems and establish all the asymptotic results using this framework. Further, we study an ordered tasks partitioning proble…
We study four problems namely, Campbell's source coding problem, Arikan's guessing problem, Huieihel et al.'s memoryless guessing problem, and Bunte and Lapidoth's task partitioning problem. We observe a close relationship among these problems. In all these problems, the objective is to minimize moments of some functions of random variables, and Rényi entropy and Sundaresan's divergence arise as optimal solutions. This motivates us to establish a connection among these four problems. In this paper, we study a more general problem and show that Rényi and Shannon entropies arise as its solution.…
The guessing number of a directed graph (digraph), equivalent to the entropy of that digraph, was introduced as a direct criterion on the solvability of a network coding instance. This paper makes two contributions on the guessing number. First, we introduce an undirected graph on all possible configurations of the digraph, referred to as the guessing graph, which encapsulates the essence of dependence amongst configurations. We prove that the guessing number of a digraph is equal to the logarithm of the independence number of its guessing graph. Therefore, network coding solvability is no mor…
A guessing wiretapper's performance on a Shannon cipher system is analyzed for a source with memory. Close relationships between guessing functions and length functions are first established. Subsequently, asymptotically optimal encryption and attack strategies are identified and their performances analyzed for sources with memory. The performance metrics are exponents of guessing moments and probability of large deviations. The metrics are then characterized for unifilar sources. Universal asymptotically optimal encryption and attack strategies are also identified for unifilar sources. Guessi…
The dynamical properties of finite dynamical systems (FDSs) have been investigated in the context of coding theoretic problems, such as network coding and index coding, and in the context of hat games, such as the guessing game and Winkler's hat game. In this paper, we relate the problems mentioned above to properties of FDSs, including the number of fixed points, their stability, and their instability. We first introduce the guessing dimension and the coset dimension of an FDS and their counterparts for digraphs. Based on the coset dimension, we then strengthen the existing equivalences betwe…
We introduce the (private) entropy of a directed graph (in a new network coding sense) as well as a number of related concepts. We show that the entropy of a directed graph is identical to its guessing number and can be bounded from below with the number of vertices minus the size of the graph's shortest index code. We show that the Network Coding solvability of each specific multiple unicast network is completely determined by the entropy (as well as by the shortest index code) of the directed graph that occur by identifying each source node with each corresponding target node. Shannon's info…
Equivocation rate has been widely used as an information-theoretic measure of security after Shannon[10]. It simplifies problems by removing the effect of atypical behavior from the system. In [9], however, Merhav and Arikan considered the alternative of using guessing exponent to analyze the Shannon's cipher system. Because guessing exponent captures the atypical behavior, the strongest expressible notion of secrecy requires the more stringent condition that the size of the key, instead of its entropy rate, to be equal to the size of the message. The relationship between equivocation and gues…
In this paper, we are interested in the number of fixed points of functions $f:A^n\to A^n$ over a finite alphabet $A$ defined on a given signed digraph $D$. We first use techniques from network coding to derive some lower bounds on the number of fixed points that only depends on $D$. We then discover relationships between the number of fixed points of $f$ and problems in coding theory, especially the design of codes for the asymmetric channel. Using these relationships, we derive upper and lower bounds on the number of fixed points, which significantly improve those given in the literature. We…
Blau and Michaeli recently introduced a novel concept for inverse problems of signal processing, that is, the perception-distortion tradeoff. We introduce their tradeoff into the rate distortion theory of variable-length lossy source coding in information theory, and clarify the tradeoff among information rate, distortion and perception for general information sources. We also discuss the fixed-length coding with average distortion criterion that was missing in the previous letter.
In this paper, we investigate the optimal tradeoff between source and channel coding for channels with bit or packet erasure. Upper and Lower bounds on the optimal channel coding rate are computed to achieve minimal end-to-end distortion. The bounds are calculated based on a combination of sphere packing, straight line and expurgated error exponents and also high rate vector quantization theory. By modeling a packet erasure channel in terms of an equivalent bit erasure channel, we obtain bounds on the packet size for a specified limit on the distortion.
Randomized network coding (RNC) greatly reduces the complexity of implementing network coding in large-scale, heterogeneous networks. This paper examines two tradeoffs in applying RNC: The first studies how the performance of RNC varies with a node's randomizing capabilities. Specifically, a limited randomized network coding (L-RNC) scheme - in which intermediate nodes perform randomized encoding based on only a limited number of random coefficients - is proposed and its performance bounds are analyzed. Such a L-RNC approach is applicable to networks in which nodes have either limited computat…
This paper addresses the fundamental characteristics of information exchange via multihop network coding over two-way relaying in a wireless ad hoc network. The end-to-end rate regions achieved by time-division multihop (TDMH), MAC-layer network coding (MLNC) and PHY-layer network coding (PLNC) are first characterized. It is shown that MLNC does not always achieve better rates than TDMH, time sharing between TDMH and MLNC is able to achieve a larger rate region, and PLNC dominates the rate regions achieved by TDMH and MLNC. An opportunistic scheduling algorithm for MLNC and PLNC is then propos…
This paper develops a diversity-multiplexing tradeoff (DMT) over a bidirectional random relay set in a wireless network where the distribution of all nodes is a stationary Poisson point process. This is a nontrivial extension of the DMT because it requires consideration of the cooperation (or lack thereof) of relay nodes, the traffic pattern and the time allocation between the forward and reverse traffic directions. We then use this tradeoff to compare the DMTs of traditional time-division multihop (TDMH) and network coding (NC). Our main results are the derivations of the DMT for both TDMH an…
Digital contents in large scale distributed storage systems may have different reliability and access delay requirements, and for this reason, erasure codes with different strengths need to be utilized to achieve the best storage efficiency. At the same time, in such large scale distributed storage systems, nodes fail on a regular basis, and the contents stored on them need to be regenerated and stored on other healthy nodes, the efficiency of which is an important factor affecting the overall quality of service. In this work, we formulate the problem of multilevel diversity coding with regene…
Network coding and cooperative communication have received considerable attention from the research community recently in order to mitigate the adverse effects of fading in wireless transmissions and at the same time to achieve high throughput and better spectral efficiency. In this work, we design and analyze deterministic and random network coding schemes for a cooperative communication setup with multiple sources and destinations. We show that our schemes outperform conventional cooperation in terms of the diversity-multiplexing tradeoff (DMT). Specifically, it achieves the full-diversity o…
We study secure source-coding with causal disclosure, under the Gaussian distribution. The optimality of Gaussian auxiliary random variables is shown in various scenarios. We explicitly characterize the tradeoff between the rates of communication and secret key. This tradeoff is the result of a mutual information optimization under Markov constraints. As a corollary, we deduce a general formula for Wyner's Common Information in the Gaussian setting.
Diversity-Multiplexing tradeoff (DMT) is a coarse high SNR approximation of the fundamental tradeoff between data rate and reliability in a slow fading channel. In this paper, we characterize the fundamental DMT of the two user single antenna Gaussian interference channel. We show that the class of multilevel superposition coding schemes universally achieves (for all fading statistics) the DMT for the two-user interference channel. For the special case of symmetric DMT, when the two users have identical rate and diversity gain requirements, we characterize the DMT achieved by the Han-Kobayashi…
It is shown that fully-parallel encoding and decoding schemes with asymptotic block error probability that scales as $O\left(f\left(n\right)\right)$ have Thompson energy that scales as $Ω\left(\sqrt{\ln f\left(n\right)}n\right)$. As well, it is shown that the number of clock cycles (denoted $T\left(n\right)$) required for any encoding or decoding scheme that reaches this bound must scale as $T\left(n\right)\ge\sqrt{\ln f\left(n\right)}$. Similar scaling results are extended to serialized computation. The Grover information-friction energy model is generalized to three dimensions and the optima…
We study the information leakage to a guessing adversary in zero-error source coding. The source coding problem is defined by a confusion graph capturing the distinguishability between source symbols. The information leakage is measured by the ratio of the adversary's successful guessing probability after and before eavesdropping the codeword, maximized over all possible source distributions. Such measurement under the basic adversarial model where the adversary makes a single guess and allows no distortion between its estimator and the true sequence is known as the maximum min-entropy leakage…
Network coding and cooperative communication have received considerable attention from the research community recently in order to mitigate the adverse effects of fading in wireless transmissions and at the same time to achieve high throughput and better spectral efficiency. In this work, we analyze a network coding scheme for a cooperative communication setup with multiple sources and destinations. The proposed protocol achieves the full diversity order at the expense of a slightly reduced multiplexing rate compared to existing schemes in the literature. We show that our scheme outperforms co…
Universal fixed-to-variable lossless source coding for memoryless sources is studied in the finite blocklength and higher-order asymptotics regimes. Optimal third-order coding rates are derived for general fixed-to-variable codes and for prefix codes. It is shown that the non-prefix Type Size code, in which codeword lengths are chosen in ascending order of type class size, achieves the optimal third-order rate and outperforms classical Two-Stage codes. Converse results are proved making use of a result on the distribution of the empirical entropy and Laplace's approximation. Finally, the fixed…
Coded distributed computing can alleviate the communication load by leveraging the redundant storage and computation resources with coding techniques in distributed computing. In this paper, we study a MapReduce-type distributed computing framework over star topological network, where all the workers exchange information through a common access point. The optimal tradeoff among the normalized number of the stored files (storage load), computed intermediate values (computation load) and transmitted bits in the uplink and downlink (communication loads) are characterized. A coded computing scheme…
This paper considers the Gaussian multiple-access channel (MAC) in the asymptotic regime where the number of users grows linearly with the code length. We propose efficient coding schemes based on random linear models with approximate message passing (AMP) decoding and derive the asymptotic error rate achieved for a given user density, user payload (in bits), and user energy. The tradeoff between energy-per-bit and achievable user density (for a fixed user payload and target error rate) is studied, and it is demonstrated that in the large system limit, a spatially coupled coding scheme with AM…
We consider the storage-retrieval rate tradeoff in private information retrieval (PIR) systems using a Shannon-theoretic approach. Our focus is mostly on the canonical two-message two-database case, for which a coding scheme based on random codebook generation and the binning technique is proposed. This coding scheme reveals a hidden connection between PIR and the classic multiple description source coding problem. We first show that when the retrieval rate is kept optimal, the proposed non-linear scheme can achieve better performance over any linear scheme. Moreover, a non-trivial storage-ret…
Side information provides a pivotal role for message delivery in many communication scenarios to accommodate increasingly large data sets, e.g., caching networks. Although index coding provides a fundamental modeling framework to exploit the benefits of side information, the index coding problem itself still remains open and only a few instances have been solved. In this paper, we propose a novel sparse and low- rank optimization modeling framework for the index coding problem to characterize the tradeoff between the amount of side information and the achievable data rate. Specifically, sparsi…
Asymptotic energy-distortion performance of zero-delay communication scenarios under additive white Gaussian noise is investigated. Using high-resolution analysis for quantizer design, the higher-order term in the logarithm of the distortion (termed the {\em energy-distortion dispersion}) is optimized while keeping the leading term (i.e., {\em energy-distortion exponent}) at its optimal value. For uniform and Gaussian sources, significant gains are observed compared to naïvely performed quantization, i.e., aimed at optimizing the source coding performance instead of the end-to-end distortion i…
The complementary problems of masking and amplifying channel state information in the Gel'fand-Pinsker channel have recently been solved by Merhav and Shamai, and Kim et al., respectively. In this paper, we study a related source coding problem. Specifically, we consider the two-encoder source coding setting where one source is to be amplified, while the other source is to be masked. In general, there is a tension between these two objectives which is characterized by the amplification-masking tradeoff. In this paper, we give a single-letter description of this tradeoff. We apply this result, …
We propose a unified coded framework for distributed computing with straggling servers, by introducing a tradeoff between "latency of computation" and "load of communication" for some linear computation tasks. We show that the coded scheme of [1]-[3] that repeats the intermediate computations to create coded multicasting opportunities to reduce communication load, and the coded scheme of [4], [5] that generates redundant intermediate computations to combat against straggling servers can be viewed as special instances of the proposed framework, by considering two extremes of this tradeoff: mini…
This work focuses on the minimum transmission energy required for communicating a pair of correlated Gaussian sources over a two-user Gaussian broadcast channel with noiseless and causal channel output feedback (GBCF). We study the fundamental limit on the required transmission energy for broadcasting a pair of source samples, such that each source can be reconstructed at its respective receiver to within a target distortion, when the source-channel bandwidth ratio is not restricted. We derive a lower bound and three distinct upper bounds on the minimum required energy. For the upper bounds we…
We study the information leakage to a guessing adversary in index coding with a general message distribution. Under both vanishing-error and zero-error decoding assumptions, we develop lower and upper bounds on the optimal leakage rate, which are based on the broadcast rate of the subproblem induced by the set of messages the adversary tries to guess. When the messages are independent and uniformly distributed, the lower and upper bounds match, establishing an equivalence between the two rates.
In this paper, we consider the problem of guessing a sequence subject to a distortion constraint. Specifically, we assume the following game between Alice and Bob: Alice has a sequence $\bx$ of length $n$. Bob wishes to guess $\bx$, yet he is satisfied with finding any sequence $\hat{\bx}$ which is within a given distortion $D$ from $\bx$. Thus, he successively submits queries to Alice, until receiving an affirmative answer, stating that his guess was within the required distortion. Finding guessing strategies which minimize the number of guesses (the \emph{guesswork}), and analyzing its prope…
We develop a framework to optimize the tradeoff between diversity, multiplexing, and delay in MIMO systems to minimize end-to-end distortion. We first focus on the diversity-multiplexing tradeoff in MIMO systems, and develop analytical results to minimize distortion of a vector quantizer concatenated with a space-time MIMO channel code. In the high SNR regime we obtain a closed-form expression for the end-to-end distortion as a function of the optimal point on the diversity-multiplexing tradeoff curve. For large but finite SNR we find this optimal point via convex optimization. We then conside…
We consider the problem of revealing/sharing data in an efficient and secure way via a compact representation. The representation should ensure reliable reconstruction of the desired features/attributes while still preserve privacy of the secret parts of the data. The problem is formulated as a remote lossy source coding with a privacy constraint where the remote source consists of public and secret parts. Inner and outer bounds for the optimal tradeoff region of compression rate, distortion, and privacy leakage rate are given and shown to coincide for some special cases. When specializing the…
The problem of multilevel diversity coding with secure regeneration is revisited. Under the assumption that the eavesdropper can access the repair data for all compromised storage nodes, Shao el al. provided a precise characterization of the minimum-bandwidth-regeneration (MBR) point of the achievable normalized storage-capacity repair-bandwidth tradeoff region. In this paper, it is shown that the MBR point of the achievable normalized storage-capacity repair-bandwidth tradeoff region remains the same even if we assume that the eavesdropper can access the repair data for some compromised stora…
To improve diversity gain in an interference channel and hence to maximize diversity multiplexing tradeoff (DMT), we propose on-off switched interference alignment (IA) where IA is intermittently utilized by switching IA on/off. For on-off switching, either IA with symbol extension or IA with Alamouti coding is adopted in this paper. Deriving and analyzing DMT of the proposed schemes, we reveal that the intermittent utilization of IA with simultaneous non-unique decoding can improve DMT in the 2-user X-channel with two antennas. Both the proposed schemes are shown to achieve diversity gain of …
The Rate-Distortion-Perception-Classification Tradeoff: Joint Source Coding and Modulation via Inverse-Domain GANs
School of Informatics, Xiamen University（厦门大学信息学院）；School of Electronics and Information Engineering, Guangxi Normal University（广西师范大学电子与信息工程学院）；School of Engineering & Physical Sciences, Heriot-Watt University（赫瑞瓦特大学工程与物理科学学院）
We consider the problem of lossy source coding with side information under a privacy constraint that the reconstruction sequence at a decoder should be kept secret to a certain extent from another terminal such as an eavesdropper, a sender, or a helper. We are interested in how the reconstruction privacy constraint at a particular terminal affects the rate-distortion tradeoff. In this work, we allow the decoder to use a random mapping, and give inner and outer bounds to the rate-distortion-equivocation region for different cases where the side information is available non-causally and causally…
Information leakage to a guessing adversary in index coding is studied, where some messages in the system are sensitive and others are not. The non-sensitive messages can be used by the server like secret keys to mitigate leakage of the sensitive messages to the adversary. We construct a deterministic linear coding scheme, developed from the rank minimization method based on fitting matrices (Bar-Yossef et al. 2011). The linear scheme leads to a novel upper bound on the optimal information leakage rate, which is proved to be tight over all deterministic scalar linear codes. We also derive a co…
Blau and Michaeli recently introduced a novel concept for inverse problems of signal processing, that is, the perception-distortion tradeoff. We introduce their tradeoff into the rate distortion theory of lossy source coding in information theory, and clarify the tradeoff among information rate, distortion and perception for general information sources.
In index coding, a server broadcasts multiple messages to their respective receivers, each with some side information that can be utilized to reduce the amount of communication from the server. Distributed index coding is an extension of index coding in which the messages are broadcast from multiple servers, each storing different subsets of the messages. In this paper, the optimal tradeoff among the message rates and the server broadcast rates, which is defined formally as the capacity region, is studied for a general distributed index coding problem. Inner and outer bounds on the capacity re…
The index coding problem is a fundamental transmission problem which occurs in a wide range of multicast networks. Network coding over a large finite field size has been shown to be a theoretically efficient solution to the index coding problem. However the high computational complexity of packet encoding and decoding over a large finite field size, and its subsequent penalty on encoding and decoding throughput and higher energy cost makes it unsuitable for practical implementation in processor and energy constraint devices like mobile phones and wireless sensors. While network coding over GF(…
The fundamental diversity-multiplexing tradeoff (DMT) of the quasi-static fading, symmetric $2$-user MIMO interference channel (IC) with channel state information at the transmitters (CSIT) and a short term average power constraint is obtained. The general case is considered where the interference-to-noise ratio (INR) at each receiver scales differently from the signal-to-noise ratio (SNR) at the receivers. The achievability of the DMT is proved by showing that a simple Han-Kobayashi coding scheme can achieve a rate region which is within a constant (independent of SNR) number of bits from a s…
Here we write in a unified fashion (using "R(P, Q, D)") the random coding exponents in channel coding and lossy source coding. We derive their explicit forms and show, that, for a given random codebook distribution Q, the channel decoding error exponent can be viewed as an encoding success exponent in lossy source coding, and the channel correct-decoding exponent can be viewed as an encoding failure exponent in lossy source coding. We then extend the channel exponents to arbitrary D, which corresponds for D > 0 to erasure decoding and for D < 0 to list decoding. For comparison, we also derive …
In order to characterize the fundamental limit of the tradeoff between the amount of cache memory and the delivery transmission rate of multiuser caching systems, various coding schemes have been proposed in the literature. These schemes can largely be categorized into two classes, namely uncoded prefetching schemes and coded prefetching schemes. While uncoded prefetching schemes in general offer order-wise optimal performance, coded prefetching schemes often have better performance at the low cache memory regime. The significant differences in the coding components between the two classes may…
The problem of broadcasting a pair of correlated Gaussian sources using optimal separate source and channel codes is studied. Considerable performance gains over previously known separate source-channel schemes are observed. Although source-channel separation yields suboptimal performance in general, it is shown that the proposed scheme is very competitive for any bandwidth compression/expansion scenarios. In particular, for a high channel SNR scenario, it can be shown to achieve optimal power-distortion tradeoff.
This paper develops an envelope-based approach to establish a link between information and queueing theory. Unlike classical, equilibrium information theory, information envelopes focus on the dynamics of sources and coders, using functions of time that bound the number of bits generated. In the limit the information envelopes converge to the average behavior and recover the entropy of a source, respectively, the average codeword length of a coder. In contrast, on short time scales and for sources with memory it is shown that large deviations from known equilibrium results occur with non-negli…
In this paper, we consider a federated learning problem over wireless channel that takes into account the coding rate and packet transmission errors. Communication channels are modelled as packet erasure channels (PEC), where the erasure probability is determined by the block length, code rate, and signal-to-noise ratio (SNR). To lessen the effect of packet erasure on the FL performance, we propose two schemes in which the central node (CN) reuses either the past local updates or the previous global parameters in case of packet erasure. We investigate the impact of coding rate on the convergen…
Inspired by the context of compressing encrypted sources, this paper considers the general tradeoff between rate, end-to-end delay, and probability of error for lossless source coding with side-information. The notion of end-to-end delay is made precise by considering a sequential setting in which source symbols are revealed in real time and need to be reconstructed at the decoder within a certain fixed latency requirement. Upper bounds are derived on the reliability functions with delay when side-information is known only to the decoder as well as when it is also known at the encoder. When th…
