arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1612.02526cs.LGcs.AIcs.CCstat.ML
收藏

短记忆预测

Prediction with a Short Memory

Stanford University(斯坦福大学);University of Washington(华盛顿大学)

Vatsal Sharan, Sham Kakade, Percy Liang, Gregory Valiant

摘要:

我们考虑的问题,预测下一个观察给定的一系列过去的观察,并考虑在何种程度上准确的预测需要复杂的算法,明确利用长期的依赖关系。也许令人惊讶的是,我们的积极结果表明,对于一个广泛的序列类,有一个算法,预测平均良好,并根据其预测只有最近的几个观察与一组简单的汇总统计过去的观察。具体来说,我们表明,对于任何分布的观察,如果过去的观察和未来的观察之间的互信息是上界的$I$,然后一个简单的马尔可夫模型在最近的$I/ε$观察获得预期的KL误差$ε$ -因此$\ell_1$错误$\sqrtε$ -相对于最佳的预测,可以访问整个过去,并知道数据生成分布。对于一个隐马尔可夫模型与$n$隐藏状态,$I$是有界的$\log n$,一个数量,不依赖于混合时间,我们表明,平凡的预测算法的基础上的经验频率的长度$O(\log n/ε)$观测窗口实现这个错误,提供的序列的长度是$d^{Ω(\log n/ε)}$,其中$d$是观察字母表的大小。我们还建立,这一结果不能得到改善,即使是类的HALOYS,在以下两个意义上:第一,对于HALOYS与$n$隐藏状态,$\log n/ε$的窗口长度是信息理论上必要的,以实现预期的$\ell_1$错误$\sqrtε$。第二,假设强反驳某类CSP的难度,估计大小为$d$的观察字母表的马尔可夫模型所需的$d^{Θ(\log n/ε)}$样本对于任何计算上易于处理的学习算法都是必要的。

英文摘要:

We consider the problem of predicting the next observation given a sequence of past observations, and consider the extent to which accurate prediction requires complex algorithms that explicitly leverage long-range dependencies. Perhaps surprisingly, our positive results show that for a broad class of sequences, there is an algorithm that predicts well on average, and bases its predictions only on the most recent few observation together with a set of simple summary statistics of the past observations. Specifically, we show that for any distribution over observations, if the mutual information between past observations and future observations is upper bounded by $I$, then a simple Markov model over the most recent $I/ε$ observations obtains expected KL error $ε$---and hence $\ell_1$ error $\sqrtε$---with respect to the optimal predictor that has access to the entire past and knows the data generating distribution. For a Hidden Markov Model with $n$ hidden states, $I$ is bounded by $\log n$, a quantity that does not depend on the mixing time, and we show that the trivial prediction algorithm based on the empirical frequencies of length $O(\log n/ε)$ windows of observations achieves this error, provided the length of the sequence is $d^{Ω(\log n/ε)}$, where $d$ is the size of the observation alphabet. We also establish that this result cannot be improved upon, even for the class of HMMs, in the following two senses: First, for HMMs with $n$ hidden states, a window length of $\log n/ε$ is information-theoretically necessary to achieve expected $\ell_1$ error $\sqrtε$. Second, the $d^{Θ(\log n/ε)}$ samples required to estimate the Markov model for an observation alphabet of size $d$ is necessary for any computationally tractable learning algorithm, assuming the hardness of strongly refuting a certain class of CSPs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑