KOALA：面向文本到图像合成的内存高效且快速扩散模型的经验教训
KOALA: Empirical Lessons Toward Memory-Efficient and Fast Diffusion Models for Text-to-Image Synthesis
AI总结：
本文通过知识蒸馏、高分辨率丰富描述数据和步长蒸馏教师模型三项实践，基于SDXL构建了KOALA-Turbo和KOALA-Lightning两种高效T2I模型，在显著降低模型规模和推理成本的同时保持生成质量，并可在8GB显存消费级GPU上生成1024px高分辨率图像。
摘要：
随着文本到图像（T2I）合成模型规模的增大，由于需要更大内存的更昂贵GPU，它们带来了更高的推理成本，这使得除了训练数据集访问受限之外，复现这些模型也变得困难。我们的研究旨在降低这些推理成本，并探索仅使用公开可用数据集和开源模型时，T2I模型的生成能力能够扩展到何种程度。为此，通过使用事实上的标准文本到图像模型Stable Diffusion XL（SDXL），我们提出了构建高效T2I模型的三项关键实践：（1）知识蒸馏：我们探索了如何将SDXL的生成能力有效蒸馏到一个高效的U-Net中，并发现自注意力是最关键的部分。（2）数据：尽管样本更少，但具有丰富描述的高分辨率图像比大量带有简短描述的低分辨率图像更为关键。（3）教师模型：步长蒸馏的教师模型使T2I模型能够减少去噪步骤。基于这些发现，我们构建了两种类型的高效文本到图像模型，称为KOALA-Turbo和KOALA-Lightning，它们使用两个紧凑的U-Net（1B和700M），将模型大小分别缩减至SDXL U-Net的54%和69%。特别是，KOALA-Lightning-700M比SDXL快4倍，同时仍保持令人满意的生成质量。此外，与SDXL不同，我们的KOALA模型可以在配备8GB显存（3060Ti）的消费级GPU上生成1024像素的高分辨率图像。我们相信，我们的KOALA模型将具有显著的实际影响，可作为SDXL的经济高效替代方案，服务于资源受限环境中的学术研究人员和普通用户。
英文摘要：
As text-to-image (T2I) synthesis models increase in size, they demand higher inference costs due to the need for more expensive GPUs with larger memory, which makes it challenging to reproduce these models in addition to the restricted access to training datasets. Our study aims to reduce these inference costs and explores how far the generative capabilities of T2I models can be extended using only publicly available datasets and open-source models. To this end, by using the de facto standard text-to-image model, Stable Diffusion XL (SDXL), we present three key practices in building an efficient T2I model: (1) Knowledge distillation: we explore how to effectively distill the generation capability of SDXL into an efficient U-Net and find that self-attention is the most crucial part. (2) Data: despite fewer samples, high-resolution images with rich captions are more crucial than a larger number of low-resolution images with short captions. (3) Teacher: Step-distilled Teacher allows T2I models to reduce the noising steps. Based on these findings, we build two types of efficient text-to-image models, called KOALA-Turbo &-Lightning, with two compact U-Nets (1B & 700M), reducing the model size up to 54% and 69% of the SDXL U-Net. In particular, the KOALA-Lightning-700M is 4x faster than SDXL while still maintaining satisfactory generation quality. Moreover, unlike SDXL, our KOALA models can generate 1024px high-resolution images on consumer-grade GPUs with 8GB of VRAMs (3060Ti). We believe that our KOALA models will have a significant practical impact, serving as cost-effective alternatives to SDXL for academic researchers and general users in resource-constrained environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
