EvalGIM:一个用于评估生成式图像模型的库
EvalGIM: A Library for Evaluating Generative Image Models
AI总结:
针对文本到图像生成模型评估缺乏统一、灵活且能提供可行结论的基准库的问题,本文提出了 EvalGIM 库,支持跨多种数据集与指标进行即插即用式评估,并通过“评估练习”提供前沿与新型的分析方法以生成可行见解。
摘要:
随着文本到图像生成模型的使用增加,用于其评估的自动基准测试方法的采用也随之增加。然而,尽管指标和数据集大量存在,却很少有统一的基准测试库能提供一个跨多种数据集和指标进行评估的框架。此外,日益强大的基准测试方法的快速引入要求评估库对新的数据集和指标保持灵活性。最后,在综合评估以提供关于模型性能的可行结论方面仍存在差距。为了实现统一、灵活且可行的评估,我们引入了 EvalGIM(发音为“EvalGym”),这是一个用于评估生成式图像模型的库。EvalGIM 广泛支持用于衡量文本到图像生成模型的质量、多样性和一致性的数据集与指标。此外,EvalGIM 的设计将用户定制的灵活性作为首要任务,并包含一个允许即插即用添加新数据集和指标的结构。为了提供可行的评估见解,我们引入了“评估练习”,以突出特定评估问题的结论。评估练习包含两种文本到图像生成模型的最先进评估方法的易用且可复现的实现:一致性-多样性-真实性 Pareto Fronts 和跨组性能差异的分解测量。EvalGIM 还包含评估练习,引入了两种用于文本到图像生成模型的新分析方法:模型排名的鲁棒性分析和跨不同提示风格的平衡评估。我们鼓励使用 EvalGIM 进行文本到图像模型探索,并邀请在 https://github.com/facebookresearch/EvalGIM/ 进行贡献。
英文摘要:
As the use of text-to-image generative models increases, so does the adoption of automatic benchmarking methods used in their evaluation. However, while metrics and datasets abound, there are few unified benchmarking libraries that provide a framework for performing evaluations across many datasets and metrics. Furthermore, the rapid introduction of increasingly robust benchmarking methods requires that evaluation libraries remain flexible to new datasets and metrics. Finally, there remains a gap in synthesizing evaluations in order to deliver actionable takeaways about model performance. To enable unified, flexible, and actionable evaluations, we introduce EvalGIM (pronounced ''EvalGym''), a library for evaluating generative image models. EvalGIM contains broad support for datasets and metrics used to measure quality, diversity, and consistency of text-to-image generative models. In addition, EvalGIM is designed with flexibility for user customization as a top priority and contains a structure that allows plug-and-play additions of new datasets and metrics. To enable actionable evaluation insights, we introduce ''Evaluation Exercises'' that highlight takeaways for specific evaluation questions. The Evaluation Exercises contain easy-to-use and reproducible implementations of two state-of-the-art evaluation methods of text-to-image generative models: consistency-diversity-realism Pareto Fronts and disaggregated measurements of performance disparities across groups. EvalGIM also contains Evaluation Exercises that introduce two new analysis methods for text-to-image generative models: robustness analyses of model rankings and balanced evaluations across different prompt styles. We encourage text-to-image model exploration with EvalGIM and invite contributions at https://github.com/facebookresearch/EvalGIM/.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。