离散贝克曼输运模型用于单步语言建模与推理

中文摘要 AI 辅助

离散扩散和流模型是自回归语言模型的一种有前景的替代方案，但将多步采样压缩为更少的步骤通常需要蒸馏一个预训练的教师模型。这使学生模型的质量受限于教师模型，并需要昂贵的两阶段训练流程。我们引入了离散贝克曼输运模型（DBTM），其构建于一个与时间无关的流之上，该流的自主输运映射可证明地将环境空间中的任意点单步携带到单纯形顶点上的一个固定点。我们表明，这一固定点性质由一个守恒方程刻画，其残差可直接从数据中最小化，从而消除了对教师流和时间条件化的需求。在此构造下，部分训练的映射对应于在有限时间处截断的流，因此生成简化为迭代一个映射直至其达到固定点。我们进一步将该映射扩展为部分上下文插值器，其中额外的函数评估充当细化步骤而非ODE积分步骤。在语言建模和推理任务上，DBTM实现了单步和少步生成，在质量和准确性上优于离散扩散和连续流基线。

英文摘要

Discrete diffusion and flow models are a promising alternative to autoregressive language models, but compressing many-step sampling into fewer steps typically requires distilling a pretrained teacher model. This caps the student at the teacher's quality and requires a costly two-stage training pipeline. We introduce Discrete Beckmann Transport Models (DBTM), built on a time-independent flow whose autonomous transport map provably carries any point in the ambient space to a fixed point on the vertices of the simplex in a single step. We show that this fixed-point property is characterized by a conservation equation whose residual can be minimized directly from data, removing the requirement for a teacher flow and time conditioning. Under this construction, a partially trained map corresponds to the flow truncated at finite time, so generation reduces to iterating one map until it reaches a fixed point. We further extend the map to a partial-context interpolant where additional function evaluations act as refinement steps rather than ODE integration steps. On language modeling and reasoning tasks, DBTM enables one- and few-step generation that improves quality and accuracy over discrete diffusion and continuous flow baselines.

发表机构

• University of Pennsylvania（宾夕法尼亚大学）
• Harvard University（哈佛大学）
• Kempner Institute（肯普纳研究所）
• IAIFI

机构由 AI 辅助整理，请以论文原文为准。