A new source model, which consists of an intrinsic state part and an extrinsic observation part, is proposed and its information-theoretic characterization, namely its rate-distortion function, is defined and analyzed. Such a source model is motivated by the recent surge of interest in the semantic aspect of information: the intrinsic state corresponds to the semantic feature of the source, which in general is not observable but can only be inferred from the extrinsic observation. There are two distortion measures, one between the intrinsic state and its reproduction, and the other between the extrinsic observation and its reproduction. Under a given code rate, the tradeoff between these two distortion measures is characterized by the rate-distortion function, which is solved via the indirect rate-distortion theory and is termed as the semantic rate-distortion function of the source. As an application of the general model and its analysis, the case of Gaussian extrinsic observation is studied, assuming a linear relationship between the intrinsic state and the extrinsic observation, under a quadratic distortion structure. The semantic rate-distortion function is shown to be the solution of a convex programming programming problem with respect to an error covariance matrix, and a reverse water-filling type of solution is provided when the model further satisfies a diagonalizability condition.
The indirect source-coding problem in which a Bernoulli process is compressed in a lossy manner from its noisy observations is considered. These noisy observations are obtained by passing the source sequence through a The indirect source-coding problem in which a Bernoulli process is compressed in a lossy manner from its noisy observations is considered. These noisy observations are obtained by passing the source sequence through a binary symmetric channel so that the channel crossover probability controls the amount of information available about the source realization at the encoder. We use …
One of the main focus in federated learning (FL) is the communication efficiency since a large number of participating edge devices send their updates to the edge server at each round of the model training. Existing works reconstruct each model update from edge devices and implicitly assume that the local model updates are independent over edge devices. In FL, however, the model update is an indirect multi-terminal source coding problem, also called as the CEO problem where each edge device cannot observe directly the gradient that is to be reconstructed at the decoder, but is rather provided …
Consider a continuous signal that cannot be observed directly. Instead, one has access to multiple corrupted versions of the signal. The available corrupted signals are correlated because they carry information about the common remote signal. The goal is to reconstruct the original signal from the data collected from its corrupted versions. The information theoretic formulation of the remote reconstruction problem assumes that the corrupted signals are uniformly sampled and the focus is on optimal compression of the samples. In this paper we revisit this problem from a sampling perspective. We…
We consider the semantic rate-distortion problem motivated by task-oriented video compression. The semantic information corresponding to the task, which is not observable to the encoder, shows impacts on the observations through a joint probability distribution. The similarities among intra-frame segments and inter-frames in video compression are formulated as side information available at both the encoder and the decoder. The decoder is interested in recovering the observation and making an inference of the semantic information under certain distortion constraints. We establish the informatio…
In this paper, the analysis of excess distortion exponent for joint source-channel coding (JSCC) in semantic-aware communication systems is presented. By introducing an unobservable semantic source, we extend the classical results by Csiszar to semantic-aware communication systems. Both upper and lower bounds of the exponent for the discrete memoryless source-channel pair are established. Moreover, an extended achievable bound of the excess distortion exponent for MIMO systems is derived. Further analysis explores how the block fading and numbers of antennas influence the exponent of semantica…
In the rate-distortion function and the Maximum Entropy (ME) method, Minimum Mutual In-formation (MMI) distributions and ME distributions are expressed by Bayes-like formulas, in-cluding Negative Exponential Functions (NEFs) and partition functions. Why do these non-probability functions exist in Bayes-like formulas? On the other hand, the rate-distortion function has three disadvantages: (1) the distortion function is subjectively defined; (2) the defi-nition of the distortion function between instances and labels is often difficult; (3) it cannot be used for data compression according to the…
Fundamental Limitation of Semantic Communications: Neural Estimation for Rate-Distortion
Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Future Network of Intelligence Institute（未来智联网络研究院）；University of Hong Kong（香港大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；State Key Laboratory of Networking and Switching Technology（网络与交换技术国家重点实验室）
In this paper we study the problem of characterizing and computing the nonanticipative rate distortion function (NRDF) for partially observable multivariate Gauss-Markov processes with hard mean squared error (MSE) distortion constraints. For the finite time horizon case, we first derive the complete characterization of this problem and its corresponding optimal realization which is shown to be a linear functional of the current time sufficient statistic of the past and current observations signals. We show that when the problem is strictly feasible, it can be computed via semidefinite program…
A rate-distortion problem motivated by the consideration of semantic information is formulated and solved. The starting point is to model an information source as a pair consisting of an intrinsic state which is not observable, corresponding to the semantic aspect of the source, and an extrinsic observation which is subject to lossy source coding. The proposed rate-distortion problem seeks a description of the information source, via encoding the extrinsic observation, under two distortion constraints, one for the intrinsic state and the other for the extrinsic observation. The corresponding s…
Task-oriented image semantic communication is a new communication paradigm, which aims to transmit semantics for artificial intelligent (AI) tasks while ignoring the reconstruction quality of the images. However, in some applications, such as autonomous driving, both image reconstruction quality and the performance of the followed AI tasks must be simultaneously considered. To tackle this challenge, this paper proposes a task-oriented semantic communication scheme with semantic reconstruction (TOSC-SR). Its main goal is to simultaneously minimize pixel-level and task-relevant semantic-level di…
This paper analyzes the fundamental limit of the strategic semantic communication problem in which a transmitter obtains a limited number of indirect observation of an intrinsic semantic information source and can then influence the receiver's decoding by sending a limited number of messages to an imperfect channel. The transmitter and the receiver can have different distortion measures and can make rational decision about their encoding and decoding strategies, respectively. The decoder can also have some side information (e.g., background knowledge and/or information obtained from previous c…
Prediction-Powered Communication with Distortion Guarantees
King’s Communications, Learning & Information Processing (KCLIP) lab within the Centre for Intelligent Information Processing Systems (CIIPS), Department of Engineering, King’s College London（国王学院通信、学习与信息处理（KCLIP）实验室，位于智能信息处理系统中心（CIIPS）内，工程系，伦敦国王学院）；Department of Electrical Engineering, Chalmers University of Technology（电子工程系，查尔姆斯理工大学）；Department of Electronic Systems, Aalborg University（电子系统系，奥尔堡大学）
Semantic Variational Bayes Based on Semantic Information G Theory for Solving Latent Variables
Intelligence Engineering and Mathematics Institute, Liaoning Technical University（辽宁工程技术大学智能工程与数学研究所）；School of Computer Engineering and Applied Mathematics, Changsha University（长沙大学计算机工程与应用数学学院）
Distortion Resilience for Goal-Oriented Semantic Communication
Department of Information Convergence Engineering, Pusan National University（信息融合工程系，釜山国立大学）；Wireless Communications Research Group, Faculty of Electrical and Electronics Engineering, Ton Duc Thang University（无线通信研究组，电子工程学院， Ton Duc Thang 大学）；College of Information Science and Electronic Engineering, Zhejiang University（信息科学与电子工程学院，浙江大学）；School of Computer Science and Statistics, Trinity College Dublin, The University of Dublin（计算机科学与统计学系，三一学院，都柏林大学）
Adaptive Source-Channel Coding for Semantic Communications
Shenzhen Future Network of Intelligence Institute（深圳未来网络智能研究所）；School of Science and Engineering（科学与工程学院）；Guangdong Provincial Key Laboratory of Future Networks of Intelligence（广东省未来网络智能重点实验室）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Department of Electrical and Electronic Engineering（电子与电气工程系）；University of Hong Kong（香港大学）；School of Information and Communication Engineering（信息与通信工程学院）；State Key Laboratory of Networking and Switching Technology（网络与交换技术国家重点实验室）；Beijing University of Posts and Telecommunications（北京邮电大学）
A Synonymous Variational Perspective on the Rate-Distortion-Perception Tradeoff
Key Laboratory of Universal Wireless Communications, Ministry of Education, Beijing University of Posts and Telecommunications（信息与通信技术联合实验室，教育部，北京邮电大学）；State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications（网络与交换技术国家重点实验室，北京邮电大学）
On the Nonasymptotic Bounds of Joint Source-Channel Coding with Hierarchical Sources
East China Normal University（华东师范大学）；Xihua University（西华大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Shanghai Jiao Tong University（上海交通大学）
