arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.05149cs.CV
收藏

多尺度局部推测解码用于图像生成

Multi-Scale Local Speculative Decoding for Image Generation

Qualcomm AI Research(高通人工智能研究)

Elia Peruzzo, Guillaume Sautière, Amirhossein Habibian

AI总结:

提出多尺度局部推测解码(MuLo-SD)框架,通过低分辨率草稿模型与高分辨率目标模型结合、局部拒绝与重采样机制,加速自回归图像生成,实现高达5倍加速并保持语义对齐和感知质量。

摘要:

自回归(AR)模型在图像合成中取得了显著成功,但其顺序性带来了严重的延迟限制。推测解码提供了一种有前景的加速途径,但现有方法受限于令牌级模糊性和缺乏空间感知。在这项工作中,我们引入了多尺度局部推测解码(MuLo-SD),一种新颖的框架,结合多分辨率草稿与空间感知验证来加速AR图像生成。我们的方法利用低分辨率草稿模型配合上采样步骤来提出候选图像令牌,然后由高分辨率目标模型并行验证。关键的是,我们引入了局部拒绝和重采样机制,通过关注空间邻域而非在第一次拒绝后进行光栅扫描重采样,从而高效纠正草稿错误。当与并行解码重采样集成时,MuLo-SD实现了显著的加速——高达$\mathbf{5 imes}$——在加速方面优于推测解码和并行解码基线,同时保持相当的语义对齐和感知质量。这些结果在MS-COCO 5k验证集上使用GenEval、DPG-Bench和FID/HPSv2进行了验证。广泛的消融实验突出了上采样设计、概率池化以及局部拒绝和重采样与邻域扩展的影响。我们的方法为图像合成中的推测解码设立了新的最先进水平,弥合了效率与保真度之间的差距。项目页面见https://qualcomm-ai-research.github.io/mulo-sd-webpage/。

英文摘要:

Autoregressive (AR) models have achieved remarkable success in image synthesis, yet their sequential nature imposes significant latency constraints. Speculative Decoding offers a promising avenue for acceleration, but existing approaches are limited by token-level ambiguity and lack of spatial awareness. In this work, we introduce Multi-Scale Local Speculative Decoding (MuLo-SD), a novel framework that combines multi-resolution drafting with spatially informed verification to accelerate AR image generation. Our method leverages a low-resolution drafter paired with an up-sampling step to propose candidate image tokens, which are then verified in parallel by a high-resolution target model. Crucially, we incorporate a local rejection and resampling mechanism, enabling efficient correction of draft errors by focusing on spatial neighborhoods rather than raster-scan resampling after the first rejection. When integrated with parallel decoding resampling, MuLo-SD achieves substantial speedups -- up to $\mathbf{5\times}$ -- outperforming both speculative decoding and parallel decoding baselines in terms of acceleration, while maintaining comparable semantic alignment and perceptual quality. These results are validated using GenEval, DPG-Bench, and FID/HPSv2 on the MS-COCO 5k validation split. Extensive ablations highlight the impact of up-sampling design, probability pooling, and local rejection and resampling with neighborhood expansion. Our approach sets a new state-of-the-art in speculative decoding for image synthesis, bridging the gap between efficiency and fidelity. Project page is available at https://qualcomm-ai-research.github.io/mulo-sd-webpage/ .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑