arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.09305cs.CV
收藏

Toffee:面向主体驱动文本到图像生成的高效百万级数据集构建

Toffee: Efficient Million-Scale Dataset Construction for Subject-Driven Text-to-Image Generation

Adobe Research(奥多比研究院);University of California, Santa Cruz(加利福尼亚大学圣克鲁兹分校)

Yufan Zhou, Ruiyi Zhang, Kaizhi Zheng, Nanxuan Zhao, Jiuxiang Gu, Zichao Wang, Xin Eric Wang, Tong Sun

AI总结:

Toffee提出无需主体级微调的高效数据集构建方法,构建了含500万图像对的首个大规模主体驱动图像编辑与生成数据集,成本大幅降低且训练模型取得竞争性结果。

摘要:

在主体驱动的文本到图像生成中,近期工作通过在包含大量图像对的合成数据集上训练模型,取得了优越的性能。在这些数据集上训练后,生成模型能够以零样本方式从任意测试图像中为特定主体生成文本对齐的图像,甚至优于那些需要在测试图像上进行额外微调的方法。然而,对于大多数研究者而言,构建此类数据集的成本高得难以承受。为了生成单个训练对,当前方法会在主体图像上微调一个预训练的文本到图像模型以捕获细粒度细节,然后使用微调后的模型基于创意文本提示为同一主体创建图像。因此,构建一个包含数百万主体的百万级大规模数据集可能需要数十万GPU小时。为了解决这一问题,我们提出了Toffee,一种用于构建主体驱动编辑和生成数据集的高效方法。具体而言,我们的数据集构建不需要任何主体级别的微调。在预训练两个生成模型之后,我们能够生成无限数量的高质量样本。我们构建了首个用于主体驱动图像编辑和生成的大规模数据集,其中包含500万图像对、文本提示和掩码。我们的数据集规模是此前最大数据集的5倍,但成本却低了数万GPU小时。为了测试所提出的数据集,我们还提出了一个既能进行主体驱动图像编辑又能进行生成的模型。只需在我们提出的数据集上训练该模型,即可获得具有竞争力的结果,这说明了所提出的数据集构建框架的有效性。

英文摘要:

In subject-driven text-to-image generation, recent works have achieved superior performance by training the model on synthetic datasets containing numerous image pairs. Trained on these datasets, generative models can produce text-aligned images for specific subject from arbitrary testing image in a zero-shot manner. They even outperform methods which require additional fine-tuning on testing images. However, the cost of creating such datasets is prohibitive for most researchers. To generate a single training pair, current methods fine-tune a pre-trained text-to-image model on the subject image to capture fine-grained details, then use the fine-tuned model to create images for the same subject based on creative text prompts. Consequently, constructing a large-scale dataset with millions of subjects can require hundreds of thousands of GPU hours. To tackle this problem, we propose Toffee, an efficient method to construct datasets for subject-driven editing and generation. Specifically, our dataset construction does not need any subject-level fine-tuning. After pre-training two generative models, we are able to generate infinite number of high-quality samples. We construct the first large-scale dataset for subject-driven image editing and generation, which contains 5 million image pairs, text prompts, and masks. Our dataset is 5 times the size of previous largest dataset, yet our cost is tens of thousands of GPU hours lower. To test the proposed dataset, we also propose a model which is capable of both subject-driven image editing and generation. By simply training the model on our proposed dataset, it obtains competitive results, illustrating the effectiveness of the proposed dataset construction framework.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑