ImagenHub：标准化条件图像生成模型的评估
ImagenHub: Standardizing the evaluation of conditional image generation models
AI总结：
针对条件图像生成模型评估条件不统一、难以公平比较的问题，提出一站式评估库ImagenHub，通过统一任务数据集、推理流水线和人工评估指标，对约30个模型开展评估并得出关键结论，将持续更新排行榜追踪领域进展。
摘要：
近年来，大量条件图像生成与编辑模型被开发出来，用于服务不同的下游任务，包括文本到图像生成、文本引导的图像编辑、主体驱动的图像生成、控制引导的图像生成等。然而，我们观察到实验条件存在巨大的不一致性：数据集、推理过程和评估指标的差异使得公平比较难以实现。本文提出了ImagenHub，这是一个一站式库，用于标准化所有条件图像生成模型的推理与评估。首先，我们定义了七项核心任务，并为这些任务整理了高质量的评估数据集。其次，我们构建了统一的推理流水线以确保公平比较。第三，我们设计了两个人工评估分数，即语义一致性和感知质量，同时制定了全面的评估指南用于评估生成的图像。我们训练了专业评分员，依据所提出的指标对模型输出进行评估。我们的人工评估在76%的模型上达到了高于0.4的Krippendorff's alpha评分者间一致性，一致性水平较高。我们共对约30个模型进行了全面评估，得出三项关键结论：（1）除文本引导的图像生成和主体驱动的图像生成外，现有模型的性能总体上不尽如人意，74%的模型总体得分低于0.5。（2）我们检验了已发表论文中的结论，发现83%的结论成立，仅有少数例外。（3）除主体驱动的图像生成外，现有自动指标的斯皮尔曼相关系数均未超过0.2。未来，我们将继续评估新发表的模型并更新排行榜，以追踪条件图像生成领域的进展。
英文摘要：
Recently, a myriad of conditional image generation and editing models have been developed to serve different downstream tasks, including text-to-image generation, text-guided image editing, subject-driven image generation, control-guided image generation, etc. However, we observe huge inconsistencies in experimental conditions: datasets, inference, and evaluation metrics - render fair comparisons difficult. This paper proposes ImagenHub, which is a one-stop library to standardize the inference and evaluation of all the conditional image generation models. Firstly, we define seven prominent tasks and curate high-quality evaluation datasets for them. Secondly, we built a unified inference pipeline to ensure fair comparison. Thirdly, we design two human evaluation scores, i.e. Semantic Consistency and Perceptual Quality, along with comprehensive guidelines to evaluate generated images. We train expert raters to evaluate the model outputs based on the proposed metrics. Our human evaluation achieves a high inter-worker agreement of Krippendorff's alpha on 76% models with a value higher than 0.4. We comprehensively evaluated a total of around 30 models and observed three key takeaways: (1) the existing models' performance is generally unsatisfying except for Text-guided Image Generation and Subject-driven Image Generation, with 74% models achieving an overall score lower than 0.5. (2) we examined the claims from published papers and found 83% of them hold with a few exceptions. (3) None of the existing automatic metrics has a Spearman's correlation higher than 0.2 except subject-driven image generation. Moving forward, we will continue our efforts to evaluate newly published models and update our leaderboard to keep track of the progress in conditional image generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
