arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.20528cs.CLcs.LG
收藏

停止-思考-自动回归:结合潜在扩散规划的语言建模

Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning

Department of Computer Science, Cornell University(计算机科学系,康奈尔大学)

Justin Lovelace, Christian Belardi, Sofian Zalouk, Adhitya Polavaram, Srivatsa Kundurthy, Kilian Q. Weinberger

AI总结:

STAR-LDM通过引入思考阶段的潜在扩散规划,实现了更高效的全局规划和生成,显著提升了语言理解和叙事连贯性任务的表现。

摘要:

Stop-Think-AutoRegress语言扩散模型(STAR-LDM)将潜在扩散规划与自回归生成相结合。不同于传统自回归语言模型仅限于逐词决策,STAR-LDM引入了一个“思考”阶段,在生成过程中暂停以通过扩散优化语义计划,再继续生成。这使得在提交离散标记之前可以在连续空间中进行全局规划。评估显示,STAR-LDM在语言理解基准上显著优于同类大小的模型,并在LLM-as-judge比较中实现了叙事连贯性和常识推理任务中超过70%的胜率。该架构还允许通过轻量级分类器进行简单控制,能够无需模型再训练即可实现属性的精细控制,同时在流畅性与控制权的权衡上优于专门化方法。

英文摘要:

The Stop-Think-AutoRegress Language Diffusion Model (STAR-LDM) integrates latent diffusion planning with autoregressive generation. Unlike conventional autoregressive language models limited to token-by-token decisions, STAR-LDM incorporates a "thinking" phase that pauses generation to refine a semantic plan through diffusion before continuing. This enables global planning in continuous space prior to committing to discrete tokens. Evaluations show STAR-LDM significantly outperforms similar-sized models on language understanding benchmarks and achieves $>70\%$ win rates in LLM-as-judge comparisons for narrative coherence and commonsense reasoning. The architecture also allows straightforward control through lightweight classifiers, enabling fine-grained steering of attributes without model retraining while maintaining better fluency-control trade-offs than specialized approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑