条件总相关与自适应并行采样的序列深度
Conditional Total Correlation and the Serial Depth of Adaptive Parallel Sampling
- Shenzhen University（深圳大学）
- National Center for Applied Mathematics Shenzhen (NCAMS)（深圳国家应用数学中心）
机构由 AI 辅助整理，请以论文原文为准。
中文总结 AI 辅助
该研究针对离散向量自适应并行采样，通过推导条件总相关与采样散度的恒等式，明确序列深度的决定因素，并在马尔可夫链等场景验证其特性，为掩码扩散模型解码提供理论支撑。
中文摘要 AI 辅助
受掩码扩散模型中并行解码的启发，我们研究离散向量的自适应并行采样：每一轮中，确定性策略基于当前已观测到的值选择未揭示的坐标，所选坐标从其精确条件边际中独立采样。近似误差用前向Kullback-Leibler散度衡量，序列深度是满足规定误差预算的最小目标平均轮数。我们的核心结果是一个精确恒等式：任意策略的散度等于其揭示轮次累积的期望条件总相关，因此条件总相关是轮内并行性的精确信息成本。该恒等式给出有限阶马尔可夫链的零误差调度，其轮次复杂度与马尔可夫阶成正比，且随序列长度对数增长；对任意固定误差预算，伯努利游走也得到匹配的对数特征；还揭示了从左到右与分层揭示顺序之间的线性-对数分离。均匀随机排列在任意固定预算下需要线性期望轮数，其硬限轮次-误差权衡是一个精确的整数组合问题，我们确定了其固定轮次渐近与联合缩放前沿。均匀平衡二进制串的深度为平方对数阶，二进制独热块的深度为平方根阶，矩形版本则实现了直至1/2的所有多项式指数。这些结果将序列深度与熵、负对数似然区分开，确立条件依赖结构为并行性的基本决定因素。对掩码扩散语言模型的实验表明，伪成本可区分已部署的解码规则，且其策略排名与自采样输出的质量高度一致。
英文摘要
Motivated by parallel decoding in masked diffusion models, we study adaptive parallel sampling of discrete vectors: in each round, a deterministic policy selects unrevealed coordinates on the basis of the values observed so far, and the selected coordinates are sampled independently from their exact conditional marginals. Approximation error is measured by forward Kullback-Leibler divergence, and serial depth is the minimum target-averaged number of rounds meeting a prescribed error budget. Our central result is an exact identity: the divergence of every policy equals the expected conditional total correlation accumulated over its reveal rounds, so conditional total correlation is the exact information cost of within-round parallelism. The identity yields zero-error schedules for finite-order Markov chains with round complexity proportional to the Markov order and logarithmic in sequence length, a matching logarithmic characterization of the Bernoulli walk at every fixed error budget, and a linear-versus-logarithmic separation between left-to-right and hierarchical reveal orders. Uniform random permutations require linearly many expected rounds at every fixed budget; their hard-cap round-error tradeoff is an exact integer-composition problem whose fixed-round asymptotics and joint-scaling frontier we determine. Uniform balanced binary strings have depth of order squared logarithm, and binary one-hot blocks have square-root depth, with rectangular versions realizing every polynomial exponent up to one half. These results separate serial depth from entropy and negative log-likelihood, and establish conditional-dependence structure as a fundamental determinant of parallelizability. Experiments with a masked diffusion language model show that the pseudo-cost distinguishes deployed decoding rules and that its policy rankings agree closely with the quality of self-sampled outputs.
