Text-to-Sticker：针对人类表情的风格定制潜扩散模型
Text-to-Sticker: Style Tailoring Latent Diffusion Models for Human Expression
AI总结：
本文提出 Style Tailoring 方法，通过联合拟合内容与风格分布微调潜扩散模型，解决贴纸生成中提示对齐与风格对齐的权衡问题，并在视觉质量、提示对齐和场景多样性上显著优于提示工程基线。
摘要：
我们引入了 Style Tailoring，这是一种在特定领域中微调潜扩散模型（LDMs）的方法，以实现高视觉质量、提示对齐和场景多样性。我们选择贴纸图像生成作为目标领域，因为这类图像与大规模 LDMs 通常生成的照片级真实样本存在显著差异。我们从一个具备较强能力的文本到图像模型（如 Emu）出发，并表明依赖提示工程使用照片级真实模型生成贴纸会导致提示对齐和场景多样性较差。为克服这些不足，我们首先在数百万张通过弱监督收集的类贴纸图像上微调 Emu，以激发多样性。接着，我们从模型生成结果中筛选出人在回路（HITL）的对齐数据集和风格数据集，并分别进行微调以改善提示对齐和风格对齐。在这些数据集上顺序微调会在更好的风格对齐与提示对齐收益之间产生权衡。为解决这一权衡，我们提出了一种名为 Style Tailoring 的新型微调方法，该方法联合拟合内容和风格分布，并实现最佳权衡。评估结果表明，与对基础 Emu 模型进行提示工程以生成贴纸相比，我们的方法将视觉质量提高了 14%，提示对齐提高了 16.2%，场景多样性提高了 15.3%。
英文摘要：
We introduce Style Tailoring, a recipe to finetune Latent Diffusion Models (LDMs) in a distinct domain with high visual quality, prompt alignment and scene diversity. We choose sticker image generation as the target domain, as the images significantly differ from photorealistic samples typically generated by large-scale LDMs. We start with a competent text-to-image model, like Emu, and show that relying on prompt engineering with a photorealistic model to generate stickers leads to poor prompt alignment and scene diversity. To overcome these drawbacks, we first finetune Emu on millions of sticker-like images collected using weak supervision to elicit diversity. Next, we curate human-in-the-loop (HITL) Alignment and Style datasets from model generations, and finetune to improve prompt alignment and style alignment respectively. Sequential finetuning on these datasets poses a tradeoff between better style alignment and prompt alignment gains. To address this tradeoff, we propose a novel fine-tuning method called Style Tailoring, which jointly fits the content and style distribution and achieves best tradeoff. Evaluation results show our method improves visual quality by 14%, prompt alignment by 16.2% and scene diversity by 15.3%, compared to prompt engineering the base Emu model for stickers generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
