PEA-Diffusion：非英文文本到图像生成中基于知识蒸馏的参数高效适配器
PEA-Diffusion: Parameter-Efficient Adapter with Knowledge Distillation in non-English Text-to-Image Generation
AI总结：
针对非英文文本到图像生成问题，提出基于知识蒸馏的即插即用语言迁移方法，仅训练6M参数的轻量级适配器并冻结UNet，即可在特定语言和通用提示评估中取得优异性能，并有效应用于跨语言下游任务。
摘要：
文本到图像扩散模型因其基于文本提示生成逼真图像的能力而闻名。然而，现有工作主要集中在英文上，缺乏对非英文文本到图像模型的支持。最常用的翻译方法无法解决与语言文化相关的生成问题，而在特定语言数据集上从头开始训练的成本过高。在本文中，我们受启发提出了一种基于知识蒸馏的简单即插即用语言迁移方法。我们只需在教师知识蒸馏下，连同小型平行数据语料库一起，训练一个仅有6M参数的轻量级类MLP参数高效适配器（PEA）。我们惊讶地发现，冻结UNet的参数仍然能在特定语言提示评估集上取得显著性能，这表明PEA能够激发原始UNet的潜在生成能力。此外，在通用提示评估集上，它 closely approaches 英文文本到图像模型的性能。此外，我们的适配器可作为插件使用，在跨语言文本到图像生成的下游任务中取得显著成果。代码将在此处提供：https://github.com/OPPO-Mente-Lab/PEA-Diffusion
英文摘要：
Text-to-image diffusion models are well-known for their ability to generate realistic images based on textual prompts. However, the existing works have predominantly focused on English, lacking support for non-English text-to-image models. The most commonly used translation methods cannot solve the generation problem related to language culture, while training from scratch on a specific language dataset is prohibitively expensive. In this paper, we are inspired to propose a simple plug-and-play language transfer method based on knowledge distillation. All we need to do is train a lightweight MLP-like parameter-efficient adapter (PEA) with only 6M parameters under teacher knowledge distillation along with a small parallel data corpus. We are surprised to find that freezing the parameters of UNet can still achieve remarkable performance on the language-specific prompt evaluation set, demonstrating that PEA can stimulate the potential generation ability of the original UNet. Additionally, it closely approaches the performance of the English text-to-image model on a general prompt evaluation set. Furthermore, our adapter can be used as a plugin to achieve significant results in downstream tasks in cross-lingual text-to-image generation. Code will be available at: https://github.com/OPPO-Mente-Lab/PEA-Diffusion
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
