GeoBlock:从扩散语言模型中的依赖几何推断块粒度
GeoBlock: Inferring Block Granularity from Dependency Geometry in Diffusion Language Models
AI总结:
GeoBlock通过分析依赖几何推断块粒度,提升扩散语言模型的并行效率与依赖一致性。
摘要:
块扩散使扩散语言模型的并行细化更高效,但其解码行为依赖于块大小。现有块大小策略依赖固定规则或启发式信号,未考虑决定哪些token可安全一起细化的依赖几何。本文提出几何视角的扩散解码:强因果顺序区域需顺序更新,语义一致区域可并行细化。GeoBlock是一种几何感知的块推断框架,直接从注意力衍生的依赖几何推断块粒度。不依赖预定义调度或局部置信启发式,GeoBlock分析跨token依赖模式以识别几何稳定的细化区域,并在解码中动态确定适当块边界。通过适应依赖几何,GeoBlock保持块扩散的并行效率,同时强制依赖一致的细化,展现自回归可靠性。GeoBlock无需额外训练,无缝集成到现有块扩散架构中。在多个基准上的广泛实验表明,GeoBlock可靠地识别几何一致的块边界,并仅用少量额外计算预算提升块扩散的准确性。
英文摘要:
Block diffusion enables efficient parallel refinement in diffusion language models, but its decoding behavior depends critically on block size. Existing block-sizing strategies rely on fixed rules or heuristic signals and do not account for the dependency geometry that determines which tokens can be safely refined together. This motivates a geometry view of diffusion decoding: \emph{regions with strong causal ordering require sequential updates, whereas semantically cohesive regions admit parallel refinement.} We introduce GeoBlock, a geometry-aware block inference framework that determines block granularity directly from attention-derived dependency geometry. Instead of relying on predefined schedules or local confidence heuristics, GeoBlock analyzes cross-token dependency patterns to identify geometrically stable refinement regions and dynamically determines appropriate block boundaries during decoding. By adapting block granularity to the dependency geometry, GeoBlock preserves the parallel efficiency of block diffusion while enforcing dependency-consistent refinement that exhibits autoregressive reliability. GeoBlock requires no additional training and integrates seamlessly into existing block diffusion architectures. Extensive experiments across multiple benchmarks show that GeoBlock reliably identifies geometry-consistent block boundaries and improves the accuracy of block diffusion with only a small additional computational budget.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。