arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.10630cs.CV

消除VAE以实现快速且高分辨率的生成细节修复

Eliminating VAE for Fast and High-Resolution Generative Detail Restoration

MMLab, ByteDance(字节实验室,字节跳动)

Yan Wang, Shijie Zhao, Junlin Li, Li Zhang

AI总结:

GenDR-Pix通过消除VAE并引入多阶段对抗蒸馏和傅里叶空间损失,实现了快速且高分辨率的生成细节修复,推理速度提升2.8倍,内存节省60%。

摘要:

扩散模型在现实世界的超分辨率(SR)任务中取得了显著突破,尽管在推理速度和设备需求上较慢。为了加快推理速度,最近的工作如GenDR采用步骤蒸馏来将步骤数减少到一个。然而,内存边界仍然限制了最大处理尺寸,需要逐块修复高分辨率图像。通过分析整个流程,我们发现变分自编码器(VAE)是延迟和内存的瓶颈。为了彻底解决问题,我们利用像素(无)插值操作来消除VAE,将基于潜在空间的GenDR反转为像素空间的GenDR-Pix。然而,使用x8像素插值进行放大可能会产生重复图案的伪影。为减轻这种失真,我们提出了一种多阶段对抗蒸馏,逐步去除编码器和解码器。具体来说,我们利用前一阶段模型的生成特征来指导对抗性判别。此外,我们提出随机填充来增强生成特征并避免判别器崩溃。我们还引入了掩码傅里叶空间损失,以惩罚幅度的异常值。为了提高推理性能,我们经验性地整合了基于填充的自增强与分类器无关的指导,以提高推理扩展性。实验结果表明,GenDR-Pix相比GenDR在推理速度上提高了2.8倍,内存节省了60%,视觉退化可忽略不计,超过了其他单步扩散SR方法。尽管面临各种困难,GenDR-Pix能够在1秒内以6GB内存恢复4K图像。

英文摘要:

Diffusion models have attained remarkable breakthroughs in the real-world super-resolution (SR) task, albeit at slow inference and high demand on devices. To accelerate inference, recent works like GenDR adopt step distillation to minimize the step number to one. However, the memory boundary still restricts the maximum processing size, necessitating tile-by-tile restoration of high-resolution images. Through profiling the pipeline, we pinpoint that the variational auto-encoder (VAE) is the bottleneck of latency and memory. To completely solve the problem, we leverage pixel-(un)shuffle operations to eliminate the VAE, reversing the latent-based GenDR to pixel-space GenDR-Pix. However, upscale with x8 pixelshuffle may induce artifacts of repeated patterns. To alleviate the distortion, we propose a multi-stage adversarial distillation to progressively remove the encoder and decoder. Specifically, we utilize generative features from the previous stage models to guide adversarial discrimination. Moreover, we propose random padding to augment generative features and avoid discriminator collapse. We also introduce a masked Fourier space loss to penalize the outliers of amplitude. To improve inference performance, we empirically integrate a padding-based self-ensemble with classifier-free guidance to improve inference scaling. Experimental results show that GenDR-Pix performs 2.8x acceleration and 60% memory-saving compared to GenDR with negligible visual degradation, surpassing other one-step diffusion SR. Against all odds, GenDR-Pix can restore 4K image in only 1 second and 6GB.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑