Travis Gagie
We present a new algorithm for dynamic prefix-free coding, based on Shannon coding. We give a simple analysis and prove a better upper bound on the length of the encoding produced than the corresponding bound for dynamic Huffman coding. We show how our algorithm can be modified for efficient length-restricted coding, alphabetic coding and coding with unequal letter costs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
继续探索
An interference alignment perspective is used to identify the simplest instances (minimum possible number of edges in the alignment graph, no more than 2 interfering messages at any destination) of index coding problems where non-Shannon information inequalities are necessary for capacity characterization. In particular, this includes the first known example of a multiple unicast (one destination per message) index coding problem where non-Shannon information inequalities are shown to be necessary. The simplest multiple unicast example has 7 edges in the alignment graph and 11 messages. The si…
Statements of Shannon's Noiseless Coding Theorem by various authors, including the original, are reviewed and clarified. Traditional statements of the theorem are often unclear as to when it applies. A new notation is introduced and the domain of application is clarified. An examination of the bounds of the Theorem leads to a new symmetric restatement. It is shown that the extended upper bound is an acheivable upper bound, giving symmetry to the theorem.The relation of information entropy to the physical entropy of Gibbs and Boltmann is illustrated. Consequently, the study of Shannon Entropy i…
Approaching capacity with low complexity is a very challenging task. In this paper, we review and compare three promising coding solutions to achieve that, which are suitable for future very high-throughput, low-complexity optical communications.
Homophonic coding is a framework to reversibly convert a message into a sequence with some target distribution. This is a promising tool to generate a codeword with a biased code-symbol distribution, which is required for capacity-achieving communication by asymmetric channels. It is known that asymptotically optimal homophonic coding can be realized by a Fixed-to-Variable (FV) length code using an interval algorithm similar to a random number generator. However, FV codes are not preferable as a component of channel codes since a decoding error propagates to all subsequent codewords. As a solu…
A common complaint about adaptive prefix coding is that it is much slower than static prefix coding. Karpinski and Nekrich recently took an important step towards resolving this: they gave an adaptive Shannon coding algorithm that encodes each character in (O (1)) amortized time and decodes it in (O (\log H)) amortized time, where $H$ is the empirical entropy of the input string $s$. For comparison, Gagie's adaptive Shannon coder and both Knuth's and Vitter's adaptive Huffman coders all use (Θ(H)) amortized time for each character. In this paper we give an adaptive Shannon coder that both enco…
Shannon's secrecy system is studied in a setting, where both the legitimate decoder and the wiretapper have access to side information sequences correlated to the source, but the wiretapper receives both the coded information and the side information via channels that are more noisy than the respective channels of the legitmate decoder, which in turn, also shares a secret key with the encoder. A single--letter characterization is provided for the achievable region in the space of five figures of merit: the equivocation at the wiretapper, the key rate, the distortion of the source reconstructio…
Starting from Shannon's celebrated 1948 channel coding theorem, we trace the evolution of channel coding from Hamming codes to capacity-approaching codes. We focus on the contributions that have led to the most significant improvements in performance vs. complexity for practical applications, particularly on the additive white Gaussian noise (AWGN) channel. We discuss algebraic block codes, and why they did not prove to be the way to get to the Shannon limit. We trace the antecedents of today's capacity-approaching codes: convolutional codes, concatenated codes, and other probabilistic coding …
Many statements from the classic information theory (the theory of Shannon's entropy) have natural counterparts in the algorithmic information theory (in the framework of Kolmogorov complexity). In this paper we discuss one simple instance of the parallelism between Shannon's and Kolmogorov's theories: we prove in the setting of Kolmogorov complexity a version of Wolf's characterization of admissible rates for the fork network.
Universal Coding for Shannon Ciphers under Side-Channel Attacks
本文研究侧信道攻击下 Shannon 密码系统的通用编码，针对任意明文分布、密钥噪声观测信道和物理测量设备，推导兼顾可靠解密与信息论安全的可达速率区域。
We consider the storage-retrieval rate tradeoff in private information retrieval (PIR) systems using a Shannon-theoretic approach. Our focus is mostly on the canonical two-message two-database case, for which a coding scheme based on random codebook generation and the binning technique is proposed. This coding scheme reveals a hidden connection between PIR and the classic multiple description source coding problem. We first show that when the retrieval rate is kept optimal, the proposed non-linear scheme can achieve better performance over any linear scheme. Moreover, a non-trivial storage-ret…
We consider transmission of stationary and ergodic sources over non-ergodic composite channels with channel state information at the receiver (CSIR). Previously we introduced alternate capacity definitions to Shannon capacity, including the capacity versus outage and the expected capacity. These generalized definitions relax the constraint of Shannon capacity that all transmitted information must be decoded at the receiver. In this work alternate end-to-end distortion metrics such as the distortion versus outage and the expected distortion are introduced to relax the constraint that a single d…
We introduce the (private) entropy of a directed graph (in a new network coding sense) as well as a number of related concepts. We show that the entropy of a directed graph is identical to its guessing number and can be bounded from below with the number of vertices minus the size of the graph's shortest index code. We show that the Network Coding solvability of each specific multiple unicast network is completely determined by the entropy (as well as by the shortest index code) of the directed graph that occur by identifying each source node with each corresponding target node. Shannon's info…
This paper begins with a discussion of integration over probability types (p-types). After doing that, the paper re-visits 3 mainstay problems of classical (non-quantum) Shannon Information Theory (SIT): source coding without distortion, channel coding, and source coding with distortion. The paper proves well-known, conventional results for each of these 3 problems. However, the proofs given for these results are not conventional. They are based on complex integration techniques (approximations obtained by applying the method of steepest descent to p-type integrals) instead of the usual delta …
Although Shannon mutual information has been widely used, its effective calculation is often difficult for many practical problems, including those in neural population coding. Asymptotic formulas based on Fisher information sometimes provide accurate approximations to the mutual information but this approach is restricted to continuous variables because the calculation of Fisher information requires derivatives with respect to the encoded variables. In this paper, we consider information-theoretic bounds and approximations of the mutual information based on Kullback--Leibler divergence and Ré…
This paper considers the problem of variable-length coding over a discrete memoryless channel (DMC) with noiseless feedback. The paper provides a stochastic control view of the problem whose solution is analyzed via a newly proposed symmetrized divergence, termed extrinsic Jensen-Shannon (EJS) divergence. It is shown that strictly positive lower bounds on EJS divergence provide non-asymptotic upper bounds on the expected code length. The paper presents strictly positive lower bounds on EJS divergence, and hence non-asymptotic upper bounds on the expected code length, for the following two codi…
Refinements and Generalizations of the Shannon Lower Bound via Extensions of the Kraft Inequality
本文通过扩展Kraft不等式，改进并推广了Shannon下界，应用于各种率失真编码场景，包括更精确的界限和个体序列下界。
A simple proof for the Shannon coding theorem, using only the Markov inequality, is presented. The technique is useful for didactic purposes, since it does not require many preliminaries and the information density and mutual information follow naturally in the proof. It may also be applicable to situations where typicality is not natural.
Revisiting Shannon's Source Coding Theorem with Distributional Uncertainty under the Nonlinear Expectation Theory
本文基于非线性期望理论，针对分布不确定的信源建立非线性信源编码定理，推广了信息熵概念，为通信网络中信源的分析提供了更全面的理论框架。
A Gallager-Type Redundancy Bound for Binary Shannon-Fano Coding
研究二进制香农 - 法诺编码冗余界，通过构建法诺树等方法，得出依赖最大源概率\(p_1\)的显式七段包络，是首个此类冗余界。
Physics-Aware, Shannon-Optimal Compression via Arithmetic Coding for Distributional Fidelity
本文提出一种基于信息论的方法，利用无损压缩评估数据集的分布一致性，通过物理指导的概率表示提供绝对的信度标准，同时提升压缩效率。
This paper considers lossy source coding of $n$-dimensional memoryless sources and shows an explicit approximation to the minimum source coding rate required to sustain the probability of exceeding distortion $d$ no greater than $ε$, which is simpler than known dispersion-based approximations. Our approach takes inspiration in the celebrated classical result stating that the Shannon lower bound to rate-distortion function becomes tight in the limit $d \to 0$. We formulate an abstract version of the Shannon lower bound that recovers both the classical Shannon lower bound and the rate-distortion…
Foundations of information theory for coding theory
本文通过信息论基础构建，探讨了编码理论中噪声信道的理论极限及最大似然解码原理。
Let $W$ be a binary-input memoryless symmetric (BMS) channel with Shannon capacity $I(W)$ and fix any $α> 0$. We construct, for any sufficiently small $δ> 0$, binary linear codes of block length $O(1/δ^{2+α})$ and rate $I(W)-δ$ that enable reliable communication on $W$ with quasi-linear time encoding and decoding. Shannon's noisy coding theorem established the \emph{existence} of such codes (without efficient constructions or decoding) with block length $O(1/δ^2)$. This quadratic dependence on the gap $δ$ to capacity is known to be best possible. Our result thus yields a constructive version o…
We prove an upper bound on the Shannon capacity of a graph via a linear programming variation. We show that our bound can outperform both the Lovász theta number and the Haemers minimum rank bound. As a by-product, we also obtain a new upper bound on the broadcast rate of Index Coding.
Shannon meets von Neumann: A Minimax Theorem for Channel Coding in the Presence of a Jammer
本文研究了在存在对抗性干扰者的情况下，由一组信道组成的信道编码问题，将其视为编码器-解码器团队与干扰者之间的零和博弈。编码器-解码器团队选择随机的编码和解码策略以最小化传输的平均错误概率，而干扰者则选择状态空间上的分布以最大化此概率。该博弈的极小极大值等同于复合信道的信道编码，我们称之为该问题的香农解。极大极小值对应于寻找具有最大最小可实现错误概率的混合信道。当极小极大和极大极小值相等时，问题被称为具有鞍点或冯·诺依曼解。尽管香农解总是存在的，但冯·诺依曼解可能不存在，因为通信团队的问题具有内在的非凸性。然而，我们证明在大块长极限下，对于所有但有限多个速率，极小极大和极大极小值会趋于相等。我们明确地将此极限值作为速率函数进行表征，并获得了极小极大和极大极小值的紧有限块长界。作为推论，我们得到了在随机码下复合信道的ε-容量的显式表达式——据我们所知，这是首次此类结果。我们的结果展示了复合信道与混合信道之间更深层次的关系，也表明传统的信息论观点，通过香农解表达，最终与通过冯·诺依曼解表达的游戏论观点一致。
Identification Codes and Post-Shannon Communication: Theory, Architectures, and Emerging Applications
本文围绕识别编码展开，它扩展了香农通信范式。介绍其理论基础，双指数增长特性带来新通信架构。探讨基于该范式的新兴架构与应用领域，强调实际用例，还展示了相关系统概念、行业观点及实现示例。
We consider transmission of discrete memoryless sources (DMSes) across discrete memoryless channels (DMCs) using variable-length lossy source-channel codes with feedback. The reliability function (optimum error exponent) is shown to be equal to $\max\{0, B(1-R(D)/C)\},$ where $R(D)$ is the rate-distortion function of the source, $B$ is the maximum relative entropy between output distributions of the DMC, and $C$ is the Shannon capacity of the channel. We show that, in this setting and in this asymptotic regime, separate source-channel coding is, in fact, optimal.
In this paper, the authors provide a weak decoding version of the traditional source coding theorem of Claude Shannon. The central bound that is obtained is \[ χ>\log_ε(2^{-n(H(X)+ε)}) \] where \[ χ=\frac{\log(k)}{n(H(X)+ε)} \] and $k$ is the number of unsupervised learning classes formed out of the non-typical source sequences. The bound leads to the conclusion that if the number of classes is high enough, the reliability function might possibly be improved. The specific regime in which this improvement might be allowable is the one in which the atypical-sequence clusters are small in size an…
New non-asymptotic random coding theorems (with error probability $ε$ and finite block length $n$) based on Gallager parity check ensemble and Shannon random code ensemble with a fixed codeword type are established for discrete input arbitrary output channels. The resulting non-asymptotic achievability bounds, when combined with non-asymptotic equipartition properties developed in the paper, can be easily computed. Analytically, these non-asymptotic achievability bounds are shown to be asymptotically tight up to the second order of the coding rate as $n$ goes to infinity with either constant o…
Shannon separation theorem lays the foundation for traditional image compression and transmission schemes, which consist of JPEG type image compression methods and the usual channel coding schemes such as Turbo and LDPC codes. One of the advantages of the separate design is that each of the two components, channel coding and source coding can be handled independently without considering the other, which is the base of decades-long technologies.
The exponentially increasing demand for data storage has been facing more and more challenges during the past years. The energy costs that it represents are also increasing, and the availability of the storage hardware is not able to follow the storage demand's trend. The short lifespan of conventional storage media -- 10 to 20 years - forces the duplication of the hardware and worsens the situation. The majority of this storage demand concerns "cold" data, data very rarely accessed but that has to be kept for long periods of time. The coding abilities of synthetic DNA, and its long durability…
Developing computationally-efficient codes that approach the Shannon-theoretic limits for communication and compression has long been one of the major goals of information and coding theory. There have been significant advances towards this goal in the last couple of decades, with the emergence of turbo codes, sparse-graph codes, and polar codes. These codes are designed primarily for discrete-alphabet channels and sources. For Gaussian channels and sources, where the alphabet is inherently continuous, Sparse Superposition Codes or Sparse Regression Codes (SPARCs) are a promising class of code…
Recently, a secrecy measure based on list-reconstruction has been proposed [2], in which a wiretapper is allowed to produce a list of $2^{mR_{L}}$ reconstruction sequences and the secrecy is measured by the minimum distortion over the entire list. In this paper, we show that this list secrecy problem is equivalent to the one with secrecy measured by a new quantity \emph{lossy-equivocation}, which is proven to be the minimum optimistic 1-achievable source coding rate (the minimum coding rate needed to reconstruct the source within target distortion with positive probability for \emph{infinitely…
The distributed remote source coding (so-called CEO) problem is studied in the case where the underlying source, not necessarily Gaussian, has finite differential entropy and the observation noise is Gaussian. The main result is a new lower bound for the sum-rate-distortion function under arbitrary distortion measures. When specialized to the case of mean-squared error, it is shown that the bound exactly mirrors a corresponding upper bound, except that the upper bound has the source power (variance) whereas the lower bound has the source entropy power. Bounds exhibiting this pleasing duality o…
In this paper, we study the information-theoretic converse for the index coding problem. We generalize the definition for the alignment chain, introduced by Maleki et al., to capture more flexible relations among interfering messages at each receiver. Based on this, we derive improved converse results for the single-server index coding problem. Compared to the maximum acyclic induced subgraph (MAIS) bound, the new bounds are always as tight and can strictly outperform the MAIS bound. They can also be useful for large problems, where the generally tighter polymatroidal bound is computationally …
Shannon Bounds for Quadratic Rate-Distortion Problems
本文综述了均方误差失真下率失真问题的 Shannon 界，重点介绍了 Berger 的技术，并作为新结果将 Gray-Wyner 网络纳入已知此类界限的设定中。
We evaluate the asymptotics of equivocations, their exponents as well as their second-order coding rates under various Rényi information measures. Specifically, we consider the effect of applying a hash function on a source and we quantify the level of non-uniformity and dependence of the compressed source from another correlated source when the number of copies of the sources is large. Unlike previous works that use Shannon information measures to quantify randomness, information or uniformity, we define our security measures in terms of a more general class of information measures--the Rényi…
Huffman coding is well known to be useful in certain decision problems involving minimizing the average number of (freely chosen) queries to determine an unknown random variable. However, in problems where the queries are more constrained, the original Huffman coding no longer works. In this paper, we proposed a general model to describe such problems and two code schemes: one is Huffman-based, and the other called GBSC (Greedy Binary Separation Coding). We proved the optimality of GBSC by induction on a binary decision tree, telling us that GBSC is at least as good as Shannon coding. We then …
A new channel coding approach was proposed in [1] for random multiple access communication over the discrete-time memoryless channel. The coding approach allows users to choose their communication rates independently without sharing the rate information among each other or with the receiver. The receiver will either decode the message or report a collision depending on whether reliable message recovery is possible. It was shown that, asymptotically as the codeword length goes to infinity, the set of communication rates supporting reliable message recovery can be characterized by an achievable …
Polynomial, or Delsarte's, method in coding theory accounts for a variety of structural results on, and bounds on the size of, extremal configurations (codes and designs) in various metric spaces. In recent works of the authors the applicability of the method was extended to cover a wider range of problems in coding and information theory. In this paper we present a general framework for the method which includes previous results as particular cases. We explain how this generalization leads to new asymptotic bounds on the performance of codes in binary-input memoryless channels and the Gaussia…
We study the effects of finite-precision representation of source's probabilities on the efficiency of classic source coding algorithms, such as Shannon, Gilbert-Moore, or arithmetic codes. In particular, we establish the following simple connection between the redundancy $R$ and the number of bits $W$ necessary for representation of source's probabilities in computer's memory ($R$ is assumed to be small): \begin{equation*} W \lesssim η\log_2 \frac{m}{R}, \end{equation*} where $m$ is the cardinality of the source's alphabet, and $η\leqslant 1$ is an implementation-specific constant. In case of…
In this paper, we study a security problem on a simple wiretap network, consisting of a source node S, a destination node D, and an intermediate node R. The intermediate node connects the source and the destination nodes via a set of noiseless parallel channels, with sizes $n_1$ and $n_2$, respectively. A message $M$ is to be sent from S to D. The information in the network may be eavesdropped by a set of wiretappers. The wiretappers cannot communicate with one another. Each wiretapper can access a subset of channels, called a wiretap set. All the chosen wiretap sets form a wiretap pattern. A …
We present a development of parts of rate-distortion theory and pattern- matching algorithms for lossy data compression, centered around a lossy version of the Asymptotic Equipartition Property (AEP). This treatment closely parallels the corresponding development in lossless compression, a point of view that was advanced in an important paper of Wyner and Ziv in 1989. In the lossless case we review how the AEP underlies the analysis of the Lempel-Ziv algorithm by viewing it as a random code and reducing it to the idealized Shannon code. This also provides information about the redundancy of th…
In this paper we consider lossless source coding for a class of sources specified by the total variational distance ball centred at a fixed nominal probability distribution. The objective is to find a minimax average length source code, where the minimizers are the codeword lengths -- real numbers for arithmetic or Shannon codes -- while the maximizers are the source distributions from the total variational distance ball. Firstly, we examine the maximization of the average codeword length by converting it into an equivalent optimization problem, and we give the optimal codeword lenghts via a w…
Universal coding of integers~(UCI) is a class of variable-length code, such that the ratio of the expected codeword length to $\max\{1,H(P)\}$ is within a constant factor, where $H(P)$ is the Shannon entropy of the decreasing probability distribution $P$. However, if we consider the ratio of the expected codeword length to $H(P)$, the ratio tends to infinity by using UCI, when $H(P)$ tends to zero. To solve this issue, this paper introduces a class of codes, termed generalized universal coding of integers~(GUCI), such that the ratio of the expected codeword length to $H(P)$ is within a constan…
A tight converse bound to channel coding rate in the finite block-length regime and under AWGN conditions was recently proposed by Polyanskiy, Poor, and Verdu (PPV). The bound is a generalization of a number of other classical results, and it was also claimed to be equivalent to Shannon's 1959 cone packing bound. Unfortunately, its numerical evaluation is troublesome even for not too large values of the block-length n. In this paper we tackle the numerical evaluation by compactly expressing the PPV converse bound in terms of non-central chi-squared distributions, and by evaluating those throug…
The unit-derived method in coding theory is shown to be a unique optimal scheme for constructing and analysing codes. In many cases efficient and practical decoding methods are produced. Codes with efficient decoding algorithms at maximal distances possible are derived from unit schemes. In particular unit-derived codes from Vandermonde or Fourier matrices are particularly commendable giving rise to mds codes of varying rates with practical and efficient decoding algorithms. For a given rate and given error correction capability, explicit codes with efficient error correcting algorithms are de…
Information-theoretic Physical Layer Security for Satellite Channels
本文从卫星信道视角回顾窃听编码及更强的无密钥无条件保密概念，分析其一般构造在典型卫星信道中的适用性，并探讨物理层安全潜力与多协议协同的系统设计启示。
In entropy coding, universal coding of integers~(UCI) is a binary universal prefix code, such that the ratio of the expected codeword length to $\max\{1, H(P)\}$ is less than or equal to a constant expansion factor $K_{\mathcal{C}}$ for any probability distribution $P$, where $H(P)$ is the Shannon entropy of $P$. $K_{\mathcal{C}}^{*}$ is the infimum of the set of expansion factors. The optimal UCI is defined as a class of UCI possessing the smallest $K_{\mathcal{C}}^{*}$. Based on prior research, the range of $K_{\mathcal{C}}^{*}$ for the optimal UCI is $2\leq K_{\mathcal{C}}^{*}\leq 2.75$. Cu…
We study a new encoding scheme for lossy source compression based on spatially coupled low-density generator-matrix codes. We develop a belief-propagation guided-decimation algorithm, and show that this algorithm allows to approach the optimal distortion of spatially coupled ensembles. Moreover, using the survey propagation formalism, we also observe that the optimal distortions of the spatially coupled and individual code ensembles are the same. Since regular low-density generator-matrix codes are known to achieve the Shannon rate-distortion bound under optimal encoding as the degrees grow, o…
We discuss the interest of escort distributions and Rényi entropy in the context of source coding. We first recall a source coding theorem by Campbell relating a generalized measure of length to the Rényi-Tsallis entropy. We show that the associated optimal codes can be obtained using considerations on escort-distributions. We propose a new family of measure of length involving escort-distributions and we show that these generalized lengths are also bounded below by the Rényi entropy. Furthermore, we obtain that the standard Shannon codes lengths are optimum for the new generalized lengths mea…
Channel symmetry properties that imply the tightness of Shannon's random coding inner bound have recently been used to determine the capacity region of discrete-memoryless two-way channels (DM-TWCs). For channels without such symmetry properties, outer bounds are often needed to estimate the capacity region. However, validating symmetry conditions and/or evaluating non-trivial outer bounds are computationally demanding, especially for channels with large input and output alphabets. In this paper, three easy-to-check conditions that identify DM-TWCs with no such symmetry properties as well as a…
This paper presents new lower and upper bounds for the optimal compression of binary prefix codes in terms of the most probable input symbol, where compression efficiency is determined by the nonlinear codeword length objective of minimizing maximum pointwise redundancy. This objective relates to both universal modeling and Shannon coding, and these bounds are tight throughout the interval. The upper bounds also apply to a related objective, that of dth exponential redundancy.
The rate regions of multilevel diversity coding systems (MDCS), a sub-class of the broader family of multi-source multi-sink networks with special structure, are investigated. After showing how to enumerate all non-isomorphic MDCS instances of a given size, the Shannon outer bound and several achievable inner bounds based on linear codes are given for the rate region of each non-isomorphic instance. For thousands of MDCS instances, the bounds match, and hence exact rate regions are proven. Results gained from these computations are summarized in key statistics involving aspects such as the suf…
While Shannon's mutual information has widespread applications in many disciplines, for practical applications it is often difficult to calculate its value accurately for high-dimensional variables because of the curse of dimensionality. This paper is focused on effective approximation methods for evaluating mutual information in the context of neural population coding. For large but finite neural populations, we derive several information-theoretic asymptotic bounds and approximation formulas that remain valid in high-dimensional spaces. We prove that optimizing the population density distrib…
We propose an information transmission scheme by a swarm of anonymous oblivious mobile robots on a graph. The swarm of robots travel from a sender vertex to a receiver vertex to transmit a symbol generated at the sender. The codeword for a symbol is a pair of an initial configuration at the sender and a set of terminal configurations at the receiver. The set of such codewords forms a code. We analyze the performance of the proposed scheme in terms of its code size and transmission delay. We first demonstrate that a lower bound of the transmission delay depends on the size of the swarm, and the…
扫码添加微信好友，提出您的宝贵建议 👇
💡 备注请填写：网站反馈
发表机构由 AI 从论文文本抽取，如发现错误，欢迎反馈修正建议。