基于Transformer的掩码扩散模型可证明学习具有潜在结构的分布
Transformer-Based Masked Diffusion Models Provably Learn Distributions with Latent Structure
AI总结：
该研究针对现有掩码扩散模型（MDMs）理论存在的维度灾难问题，提出基于Transformer的MDMs，通过隐马尔可夫树混合模型建模潜在结构，证明其可自动适应未知潜在结构，以Õ(LK_par/ε)样本复杂度实现ε精度采样，为MDMs的经验有效性提供理论支撑。
摘要：
掩码扩散模型（MDMs）通过使用学习到的掩码预测器并行迭代地解封多个token来生成离散数据，其输出存在两类误差：由并行采样导致的并行化误差，以及由有限样本学习的掩码预测器产生的估计误差。现有理论将这些误差分开处理，且可用的统计界通常随词汇表大小为V、长度为L的序列数V^L缩放，存在维度灾难问题。本研究建立了端到端的统计保证，表明基于Transformer的MDMs可学习具有潜在结构的分布，其样本复杂度由该潜在结构决定。具体而言，我们通过隐马尔可夫树混合模型对该结构进行建模，该模型捕捉组件内的潜在依赖关系和组件间的异质性。当数据分布可被此类混合模型良好近似时，采用并行解封调度的基于Transformer的MDMs，在KL散度下实现ε精度采样所需的训练样本数为Õ(LK_par/ε)，其中K_par是L、V、隐状态数量及混合组件数量的多项式函数。Transformer网络和采样过程无需知晓树拓扑或组件参数，可自动适应未知的潜在结构。总体而言，我们的结果为MDMs在真实世界数据上的经验有效性提供了理论依据。
英文摘要：
Masked diffusion models (MDMs) generate discrete data by iteratively unmasking multiple tokens in parallel using a learned mask predictor. Their output carries two errors: parallelization error due to parallel sampling, and estimation error from the mask predictor learned with finite samples. Existing theory treats these errors separately, and the available statistical bounds typically scale with the number $V^L$ of length-$L$ sequences over a vocabulary of size $V$, suffering from the curse of dimensionality. In this work, we establish end-to-end statistical guarantees showing that transformer-based MDMs learn structured distributions with a sample complexity governed by their latent structure. Specifically, we model this structure through mixtures of hidden Markov trees, which capture latent dependencies within components and heterogeneity across components. When the data distribution is well approximated by such a mixture, a transformer-based MDM with parallel unmasking schedules achieves $\varepsilon$-accurate sampling in KL divergence with $\widetilde{O}(LK_{\mathrm{par}}/\varepsilon)$ training samples, where $K_{\mathrm{par}}$ is polynomial in $L$, $V$, the number of hidden states, and the number of mixture components. The transformer network and sampling procedure require no knowledge of the tree topologies or component parameters, thereby adapting to the unknown latent structure automatically. Overall, our results provide theoretical justification for the empirical effectiveness of MDMs on real-world data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
