发表机构
University of Pennsylvania（宾夕法尼亚大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文将扩散、AR等模型视为corruption格上的路径，定义调度代价，提出用预训练权重的成对依赖核预测解码调度排名，为各类生成模型解码提供设计原则。
AI 中文摘要
扩散模型与自回归（AR）长期以来被视为不同类别的生成模型，扩散模型擅长连续域，AR模型擅长离散 token。近期研究试图结合两者优势，每种混合模型都通过设计确定解码调度。本文探究能否在固定步数解码前预测某一模型解码调度的性能。我们将扩散模型、AR模型及介于两者之间的模型描述为一条 corruption 格上的路径，定义调度的代价为其并行步骤所丢弃的依赖关系。该代价表明，零代价调度的最少步数由数据的几何结构决定，对 token 和连续域均如此。具体而言，对于图上马尔可夫且沿路径依赖的数据，最少步数等于图的树深度，其随序列长度呈对数变化，随网格边长呈线性变化。步数少于树深度时，每个调度都会产生正代价，我们利用预训练权重估计的成对依赖核，可在解码前预测该代价的排名。在文本生成、图像生成和视频生成任务中，我们验证了不同调度在不同指标和基准下排名的多数预测结果。因此，本研究为未来的 AR 模型、扩散模型及介于两者之间的模型的解码提供了设计原则，代码可在该 https URL 获取。
英文摘要
Diffusion and autoregression (AR) have long been seen as different categories of generative models, with diffusion specialising in continuous fields and AR specialising in discrete tokens. Recent work seeks to combine the advantages of the two models, and each hybrid fixes its decoding schedule by design. In this paper, we ask whether the performance of decoding schedules of one model can be predicted before decoding at a fixed number of steps. We describe diffusion, AR, and models in between as paths on one corruption lattice, and define the cost of a schedule as the dependence its parallel steps discard. The cost shows that the fewest steps of a zero-cost schedule are set by the geometry of the data, in the same way for tokens and for continuous fields. In particular, for data that are Markov on a graph and dependent along its paths, the fewest steps equal the graph's treedepth, which is logarithmic in the length of a sequence and linear in the side length of a grid. With fewer steps than the treedepth, every schedule pays a positive cost, whose ranking we predict before decoding with a kernel of pairwise dependence estimated from pretrained weights. Across text generation, image generation, and video generation, we verify most of the predictions about the rankings of different schedules under different metrics and benchmarks. This work therefore provides a design principle for decoding for future AR models, diffusion models, and anything in between. Our code is available at https://github.com/TSUITUENYUE/The-Lattice-of-Transition-Laws.
