arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.16379cs.CV
收藏

FlashEval:面向文本到图像扩散生成模型的快速准确评估

FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models

Microsoft Research(微软研究院);Tsinghua University(清华大学);Infinigence-AI(智源人工智能);Shanghai Jiao Tong University(上海交通大学)

Lin Zhao, Tianchen Zhao, Zinan Lin, Xuefei Ning, Guohao Dai, Huazhong Yang, Yu Wang

AI总结:

为解决文本到图像扩散模型评估成本高的问题,提出迭代搜索算法 FlashEval,通过选取代表性数据子集实现 10 倍评估加速且保持评估质量。

摘要:

近年来,文本到图像生成模型的开发取得了显著进展。评估生成模型的质量是开发过程中的一个关键步骤。然而,评估过程会消耗大量计算资源,使得所需的模型性能周期性评估(如监控训练进度)变得不切实际。因此,我们希望通过选择文本-图像数据集的代表性子集来提升评估效率。我们系统地研究了设计选择,包括选择标准(文本特征或基于图像的指标)和选择粒度(提示级或集合级)。我们发现,先前关于训练数据子集选择的经验无法推广到这一问题中,因此提出了 FlashEval,这是一种专为评估数据选择定制的迭代搜索算法。我们在 COCO 和 DiffusionDB 数据集上证明了 FlashEval 在对具有不同配置(包括架构、量化级别和采样器调度)的扩散模型进行排序时的有效性。我们搜索得到的 50 项子集在未见过的模型上,能够达到与随机采样的 500 项 COCO 标注子集相当的评估质量,实现了 10 倍的评估加速。我们发布了这些常用数据集的浓缩子集,以帮助促进扩散算法的设计与评估,并开源了 FlashEval 作为压缩未来数据集的工具,访问地址为 https://github.com/thu-nics/FlashEval。

英文摘要:

In recent years, there has been significant progress in the development of text-to-image generative models. Evaluating the quality of the generative models is one essential step in the development process. Unfortunately, the evaluation process could consume a significant amount of computational resources, making the required periodic evaluation of model performance (e.g., monitoring training progress) impractical. Therefore, we seek to improve the evaluation efficiency by selecting the representative subset of the text-image dataset. We systematically investigate the design choices, including the selection criteria (textural features or image-based metrics) and the selection granularity (prompt-level or set-level). We find that the insights from prior work on subset selection for training data do not generalize to this problem, and we propose FlashEval, an iterative search algorithm tailored to evaluation data selection. We demonstrate the effectiveness of FlashEval on ranking diffusion models with various configurations, including architectures, quantization levels, and sampler schedules on COCO and DiffusionDB datasets. Our searched 50-item subset could achieve comparable evaluation quality to the randomly sampled 500-item subset for COCO annotations on unseen models, achieving a 10x evaluation speedup. We release the condensed subset of these commonly used datasets to help facilitate diffusion algorithm design and evaluation, and open-source FlashEval as a tool for condensing future datasets, accessible at https://github.com/thu-nics/FlashEval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑