arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.04854cs.CLcs.AIcs.LG
收藏

投影自回归:在连续状态空间中进行自回归语言生成

Projected Autoregression: Autoregressive Language Generation in Continuous State Space

IBM Research(IBM研究院)

Oshri Naparstek

AI总结:

本文提出投影自回归方法,通过连续预测和离散投影实现语言生成,展示连续状态空间在生成文本结构和动态方面的独特优势。

摘要:

标准自回归语言模型通过重复选择离散下一个标记来生成文本,每一步都耦合了预测与不可逆的承诺。我们证明,标记选择并非唯一可行的自回归接口。投影自回归用嵌入空间中的连续预测替代标记选择,在承诺时间进行离散投影。模型通过回归和对比目标预测下一个标记向量,而离散标记仅在投影时出现。可选的可变后缀("液态尾部")允许在承诺前进行迭代细化,但核心变化更基本:下一步预测是连续的,离散标记仅作为下游接口产生。投影自回归建立了一个具体的替代方案:语言生成可以围绕连续状态预测和延迟离散承诺展开。细化仍局限于左到右因果过程中的短因果后缀,而非序列范围的去噪过程。这种分离有两个后果。首先,它诱导了一种不同的生成模式:即使使用立即投影(K=1),连续预测产生的文本结构和动态也不同于测试的标记空间AR基线,包括一个计算匹配的最佳16个重排基线。其次,它暴露了自回归生成中的连续控制表面:方向速率、历史噪声、延迟承诺、状态空间引导和嵌入几何直接作用于生成状态,而在标记承诺之前。这些结果将重复标记选择置于更广泛的自回归接口家族中,并将连续状态空间作为更广泛的语言生成算法设计空间暴露出来。

英文摘要:

Standard autoregressive language models generate text by repeatedly selecting a discrete next token, coupling prediction with irreversible commitment at every step. We show that token selection is not the only viable autoregressive interface. \textbf{Projected Autoregression} replaces token selection with continuous prediction in embedding space followed by discrete projection at commitment time. The model predicts next-token vectors via regression and contrastive objectives, while discrete tokens arise only by nearest-neighbor projection. An optional mutable suffix (``liquid tail'') enables iterative refinement before commitment, but the central change is more basic: next-step prediction is continuous, and discrete tokens are produced only as a downstream interface. Projected Autoregression establishes a concrete alternative to token-selection autoregression: language generation can be organized around continuous-state prediction with delayed discrete commitment. Refinement remains local to a short causal suffix within a left-to-right causal process, rather than a sequence-wide denoising process. This separation has two consequences. First, it induces a \emph{distinct generation regime}: even with immediate projection ($K{=}1$), continuous prediction yields text structure and dynamics that differ from tested token-space AR baselines, including a compute-matched best-of-16 reranking baseline. Second, it exposes a \emph{continuous control surface} inside autoregressive generation: direction rate, history noise, delayed commitment, state-space guidance, and embedding geometry act directly on the evolving generative state before token commitment. Taken together, these results place repeated token selection within a larger family of autoregressive interfaces and expose continuous state space as a broader algorithmic design space for language generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑