任意阶GPT作为掩码扩散模型：解耦公式与架构
Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture
AI总结：
本研究通过在仅解码器框架内评估掩码扩散模型（MDMs），解耦算法与架构因素，对比MDM与自回归范式的权衡，实现约25倍推理加速，为高效基础模型开发提供见解。
摘要：
高效扩展大型语言模型（LLMs）需探索主流自回归（AR）方法的替代方案，掩码扩散模型（MDMs）成为候选。但AR（通常为仅解码器）与MDM（通常为仅编码器）范式的比较因架构差异而混淆，掩盖了真正的算法与效率权衡。本研究通过在仅解码器框架内评估MDMs来解耦这些因素，以：（1）通过阶数差异公平比较MDM（作为任意阶AR）与标准AR范式；（2）研究MDM架构对计算效率的影响。我们表明，仅解码器MDM虽建模空间更大，但结合温度退火等技术可实现约25倍的推理加速与相当的困惑度，为减少推理计算提供了途径。本研究通过将核心建模选择与架构影响解耦，为开发更具计算效率的基础模型提供了见解，代码可在该https URL获取。
英文摘要：
Efficiently scaling Large Language Models (LLMs) necessitates exploring alternatives to dominant autoregressive (AR) methods, with Masked Diffusion Models (MDMs) emerging as candidates. However, comparing AR (typically decoder-only) and MDM (often encoder-only) paradigms is confounded by differing architectures, obscuring true algorithmic and efficiency trade-offs. This research decouples these factors by evaluating MDMs within a decoder-only framework to: (1) Equitably compare MDM (as Any-Order AR) and standard AR paradigms through discrepancies on orders. (2) Investigate MDM architectural impacts on computational efficiency. We show decoder-only MDMs, despite a larger modeling space, can achieve significant inference speedups ($\sim25\times$) and comparable perplexity with techniques like temperature annealing, offering a path to reduced inference compute. This work provides insights for developing more computationally efficient foundation models by disentangling core modeling choices from architectural influences. Code is available at https://github.com/scxue/AO-GPT-MDM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
