arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.12465cs.LGcs.DS
收藏

下一 token 预测壁垒处的计算-统计权衡:错误设定下的自回归与模仿学习

Computational-Statistical Tradeoffs at the Next-Token Prediction Barrier: Autoregressive and Imitation Learning under Misspecification

Microsoft(微软)

Dhruv Rohatgi, Adam Block, Audrey Huang, Akshay Krishnamurthy, Dylan J. Foster

AI总结:

本文研究错误设定下自回归下一 token 预测的误差放大,证明其存在 $\Omega(H)$ 的内在壁垒,并揭示信息论、计算与统计之间的近似因子权衡,结论推广至模仿学习。

摘要:

使用对数损失的下一 token 预测是自回归序列建模的基石,但在实践中会受到误差放大的困扰:模型中的误差不断复合,生成质量随序列长度 $H$ 增加而下降。从理论角度看,这种现象不应出现在良好设定的环境中;事实上,越来越多的实证工作假设,错误设定——即学习器的表达能力不足以表示目标分布——可能是根本原因。在错误设定下,目标是学习到与同类最佳模型相当的效果,至多相差一个乘性近似因子 $C\geq 1$;我们证实,对于下一 token 预测,$C$ 确实随 $H$ 增长,从而为这一实证假设提供理论支持。随后,我们追问这种误差放大模式是否能在算法、计算或信息论层面避免,并揭示了内在的计算-统计权衡。我们证明:(1)在信息论意义上,可以避免误差放大并实现 $C=O(1)$。(2)下一 token 预测可以被增强为稳健的,从而实现 $C=\tilde O(H)$,代表中等程度的误差放大;但这是一道内在壁垒:任何下一 token 预测式目标都必然承受 $C=\Omega(H)$。(3)对于自回归线性模型这一天然测试平台,不存在计算高效的算法能够实现次多项式近似因子 $C=e^{(\log H)^{1-\Omega(1)}}$;然而,至少对于二元 token 空间,可以在计算能力与统计能力之间平滑权衡,并在次指数时间内改进 $C=\Omega(H)$。我们的结果对更一般的模仿学习环境也有影响,其中广泛使用的行为克隆算法正是下一 token 预测的推广。

英文摘要:

Next-token prediction with the logarithmic loss is a cornerstone of autoregressive sequence modeling, but, in practice, suffers from error amplification, where errors in the model compound and generation quality degrades as sequence length $H$ increases. From a theoretical perspective, this phenomenon should not appear in well-specified settings, and, indeed, a growing body of empirical work hypothesizes that misspecification, where the learner is not sufficiently expressive to represent the target distribution, may be the root cause. Under misspecification -- where the goal is to learn as well as the best-in-class model up to a multiplicative approximation factor $C\geq 1$ -- we confirm that $C$ indeed grows with $H$ for next-token prediction, lending theoretical support to this empirical hypothesis. We then ask whether this mode of error amplification is avoidable algorithmically, computationally, or information-theoretically, and uncover inherent computational-statistical tradeoffs. We show: (1) Information-theoretically, one can avoid error amplification and achieve $C=O(1)$. (2) Next-token prediction can be made robust so as to achieve $C=\tilde O(H)$, representing moderate error amplification, but this is an inherent barrier: any next-token prediction-style objective must suffer $C=Ω(H)$. (3) For the natural testbed of autoregressive linear models, no computationally efficient algorithm can achieve sub-polynomial approximation factor $C=e^{(\log H)^{1-Ω(1)}}$; however, at least for binary token spaces, one can smoothly trade compute for statistical power and improve on $C=Ω(H)$ in sub-exponential time. Our results have consequences in the more general setting of imitation learning, where the widely-used behavior cloning algorithm generalizes next-token prediction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑