arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.01938cs.ITmath.ITmath.STstat.MLstat.TH
收藏

Good-Turing 估计对马尔可夫样本的效果如何?

How good is Good-Turing for Markov samples?

Prafulla Chandra, Andrew Thangaraj, Nived Rajaraman

AI总结:

本文研究 Good-Turing 估计器在马尔可夫样本缺失质量估计中的收敛性,证明其取决于转移矩阵删列后的 Perron 特征值与平稳分布的关系,并首次给出秩为 2 马尔可夫链的极小极大速率上下界。

摘要:

Good-Turing(GT)估计器用于估计 n 个样本中的缺失质量(即缺失符号的总概率),其值为恰好出现一次的符号数除以 n。对于独立同分布样本,通过对所有符号的期望误差进行一致上界控制,可以证明 GT 估计器的偏差和平方误差风险以 1/n 的速率下降。本文研究 GT 估计器对马尔可夫样本的缺失平稳质量(即缺失符号的总平稳概率)的收敛性,其中字母表为 X,平稳分布为 [π_x : x ∈ X],转移概率矩阵(t.p.m.)为 P。这是一个重要且有趣的问题,因为 GT 被广泛应用于具有时间依赖性的场景中,例如为词序列分配概率的语言模型,而这些模型通常被建模为马尔可夫过程。我们证明 GT 的收敛性取决于 (P^{~x})^n 的收敛性,其中 P^{~x} 是将 P 的第 x 列置零后得到的矩阵。这又取决于 P^{~x} 的 Perron 特征值 λ^{~x} 及其与 π_x 在 x 上的一致关系。对于随机生成的 t.p.m. 以及从《纽约时报》和查尔斯·狄更斯语料库中推导出的 t.p.m.,我们通过数值实验展示了 λ^{~x} 与 π_x 之间在 x 上一致的这种关系。这支持了 GT 在语言模型和实际文本数据场景中所观察到的成功表现。对于具有谱隙 β 的秩为 2、可对角化的 t.p.m. 的马尔可夫链,我们证明了估计平稳缺失质量的极小极大速率上界和下界分别为 1/(nβ^5) 和 1/(nβ)。这一理论结果将独立同分布或秩为 1 的 t.p.m. 的 1/n 极小极大速率推广到了秩为 2 的马尔可夫情形,并且是马尔可夫样本缺失质量的首个此类极小极大速率结果。

英文摘要:

The Good-Turing (GT) estimator for the missing mass (i.e., total probability of missing symbols) in $n$ samples is the number of symbols that appeared exactly once divided by $n$. For i.i.d. samples, the bias and squared-error risk of the GT estimator can be shown to fall as $1/n$ by bounding the expected error uniformly over all symbols. In this work, we study convergence of the GT estimator for missing stationary mass (i.e., total stationary probability of missing symbols) of Markov samples on an alphabet $\mathcal{X}$ with stationary distribution $[π_x:x \in \mathcal{X}]$ and transition probability matrix (t.p.m.) $P$. This is an important and interesting problem because GT is widely used in applications with temporal dependencies such as language models assigning probabilities to word sequences, which are modelled as Markov. We show that convergence of GT depends on convergence of $(P^{\sim x})^n$, where $P^{\sim x}$ is $P$ with the $x$-th column zeroed out. This, in turn, depends on the Perron eigenvalue $λ^{\sim x}$ of $P^{\sim x}$ and its relationship with $π_x$ uniformly over $x$. For randomly generated t.p.ms and t.p.ms derived from New York Times and Charles Dickens corpora, we numerically exhibit such uniform-over-$x$ relationships between $λ^{\sim x}$ and $π_x$. This supports the observed success of GT in language models and practical text data scenarios. For Markov chains with rank-2, diagonalizable t.p.ms having spectral gap $β$, we show minimax rate upper and lower bounds of $1/(nβ^5)$ and $1/(nβ)$, respectively, for the estimation of stationary missing mass. This theoretical result extends the $1/n$ minimax rate for i.i.d. or rank-1 t.p.ms to rank-2 Markov, and is a first such minimax rate result for missing mass of Markov samples.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑