Bounds on the entropy of patterns of sequences generated by independently identically distributed (i.i.d.) sources are derived. A pattern is a sequence of indices that contains all consecutive integer indices in increasing order of first occurrence. If the alphabet of a source that generated a sequence is unknown, the inevitable cost of coding the unknown alphabet symbols can be exploited to create the pattern of the sequence. This pattern can in turn be compressed by itself. The bounds derived here are functions of the i.i.d. source entropy, alphabet size, and letter probabilities. It is shown that for large alphabets, the pattern entropy must decrease from the i.i.d. one. The decrease is in many cases more significant than the universal coding redundancy bounds derived in prior works. The pattern entropy is confined between two bounds that depend on the arrangement of the letter probabilities in the probability space. For very large alphabets whose size may be greater than the coded pattern length, all low probability letters are packed into one symbol. The pattern entropy is upper and lower bounded in terms of the i.i.d. entropy of the new packed alphabet. Correction terms, which are usually negligible, are provided for both upper and lower bounds.
Thermal jitter (phase noise) from a free-running ring oscillator is a common, easily implementable physical randomness source in True Random Number Generators (TRNGs). We show how to evaluate entropy, autocorrelation, and bit pattern distributions of ring oscillator noise sources, even with low jitter levels or some bias. Entropy justification is required in NIST 800-90B and AIS-31 testing and for applications such as the RISC-V entropy source extension. Our numerical evaluation algorithms outperform Monte Carlo simulations in speed and accuracy. We also propose a new lower bound estimation fo…
In this note, we studied the asymptotic behaviour of the length of the longest common substring for run-length encoded sequences. When the original sequences are generated by an $α$-mixing process with exponential decay (or $ψ$-mixing with polynomial decay), we proved that this length grows logarithmically with a coefficient depending on the Rényi entropy of the pushforward measure. For Bernoulli processes and Markov chains, this coefficient is computed explicitly.
In this work we implement the so-called matching time estimators for estimating the entropy rate as well as the entropy production rate for symbolic sequences. These estimators are based on recurrence properties of the system, which have been shown to be appropriate to test irreversibility specially when the sequences have large correlations or memory. Based on limit theorems for matching-times we derive a maximum likelihood estimator for entropy rate assuming that we have a set of moderately short symbolic time-series of finite random duration. We show that the proposed estimator has several …
Universidade Federal de Minas Gerais（米纳斯吉拉斯联邦大学）；Victoria University of Wellington（惠灵顿维多利亚大学）；Instituto Tecnológico de Buenos Aires（布宜诺斯艾利斯技术研究所）；Universidad Nacional de Tres de Febrero（二月三日国立大学）；Universidad Nacional del Nordeste（东北国立大学）；CONICET（阿根廷国家科学与技术研究理事会）；Universidad Tecnológica Nacional（国立技术大学）
The ultimate purpose of the statistical analysis of ordinal patterns is to characterize the distribution of the features they induce. In particular, knowing the joint distribution of the pair Entropy-Statistical Complexity for a large class of time series models would allow statistical tests that are unavailable to date. Working in this direction, we characterize the asymptotic distribution of the empirical Shannon's Entropy for any model under which the true normalized Entropy is neither zero nor one. We obtain the asymptotic distribution from the Central Limit Theorem (assuming large time se…
Dictionary Based Pattern Entropy for Causal Direction Discovery
Department of Computer Science and Information Systems（计算机科学与信息系统系）；BITS Pilani K K Birla Goa Campus（比斯潘尼学院K K Birla Goa校区）；Complex Systems Programme（复杂系统计划）；National Institute of Advanced Studies（国家高级研究所）；Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
The discovery of nucleic acids and the structure of DNA have brought considerable advances in the understanding of life. The development of next-generation sequencing technologies has led to a large-scale generation of data, for which computational methods have become essential for analysis and knowledge discovery. In particular, RNAs have received much attention because of the diversity of their functionalities in the organism and the discoveries of different classes with different functions in many biological processes. Therefore, the correct identification of RNA sequences is increasingly i…
