arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.09341cs.CVcs.AIcs.DB
收藏

人类偏好评分 v2:一个用于评估文本到图像合成中人类偏好的可靠基准

Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis

The Chinese University of Hong Kong(香港中文大学);SenseTime Research(商汤科技研究院);Qing Yuan Research Institute, Shanghai Jiao Tong University(上海交通大学清源研究院);Centre for Perceptual and Interactive Intelligence(感知与交互智能研究中心)

Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, Hongsheng Li

AI总结:

本文提出大规模人类偏好数据集 HPD v2 并微调 CLIP 得到评分模型 HPS v2,用于更准确地评估文本到图像生成模型,并建立了相关基准。

摘要:

近年来,文本到图像生成模型能够根据文本输入生成高保真图像,但现有评估指标无法准确评价这些生成图像的质量。为解决这一问题,我们引入了人类偏好数据集 v2(HPD v2),这是一个大规模数据集,采集了来自广泛来源的图像上的人类偏好。HPD v2 包含 433,760 对图像上的 798,090 条人类偏好选择,是同类数据集中规模最大的。文本提示和图像经过刻意收集以消除潜在偏差,而这是以往数据集中常见的问题。通过在 HPD v2 上微调 CLIP,我们获得了人类偏好评分 v2(HPS v2),这是一个能够更准确地预测生成图像上人类偏好的评分模型。我们的实验表明,HPS v2 在各种图像分布上比以往的指标具有更好的泛化能力,并且对文本到图像生成模型的算法改进具有响应性,使其成为这些模型的更优评估指标。我们还研究了文本到图像生成模型评估提示的设计,以使评估稳定、公平且易于使用。最后,我们利用 HPS v2 建立了一个文本到图像生成模型基准,其中包含来自学术界、社区和工业界的一组近期文本到图像模型。代码和数据集可在 https://github.com/tgxs002/HPSv2 获取。

英文摘要:

Recent text-to-image generative models can generate high-fidelity images from text inputs, but the quality of these generated images cannot be accurately evaluated by existing evaluation metrics. To address this issue, we introduce Human Preference Dataset v2 (HPD v2), a large-scale dataset that captures human preferences on images from a wide range of sources. HPD v2 comprises 798,090 human preference choices on 433,760 pairs of images, making it the largest dataset of its kind. The text prompts and images are deliberately collected to eliminate potential bias, which is a common issue in previous datasets. By fine-tuning CLIP on HPD v2, we obtain Human Preference Score v2 (HPS v2), a scoring model that can more accurately predict human preferences on generated images. Our experiments demonstrate that HPS v2 generalizes better than previous metrics across various image distributions and is responsive to algorithmic improvements of text-to-image generative models, making it a preferable evaluation metric for these models. We also investigate the design of the evaluation prompts for text-to-image generative models, to make the evaluation stable, fair and easy-to-use. Finally, we establish a benchmark for text-to-image generative models using HPS v2, which includes a set of recent text-to-image models from the academic, community and industry. The code and dataset is available at https://github.com/tgxs002/HPSv2 .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑