arXiv 2406.137432024-11-03cs.CVcs.AIcs.CLcs.LGcs.MM

GenAI-Bench：评估并改进组合式文本到视觉生成

GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation

Carnegie Mellon University（卡内基梅隆大学）；Meta

Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, Deva Ramanan

备注：: We open-source our dataset, model, and code at: https://linzhiqiu.github.io/papers/genai_bench ; Project page: https://linzhiqiu.github.io/papers/genai_bench ; GenAI-Bench was first introduced in arxiv:2404.01291. This article extends it with an additional GenAI-Rank benchmark

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

该研究基于GenAI-Bench评估文本到视觉模型的组合生成能力，发现VQAScore指标优于现有方法，可通过黑盒排序提升生成效果，并发布含超8万条人类评分的基准以推动相关研究。

摘要：

尽管当前文本到视觉模型已能生成照片级真实感的图像和视频，但它们在处理涉及属性、关系以及逻辑、比较等高阶推理的组合式文本提示时仍存在困难。本研究针对GenAI-Bench开展了大规模人类评估研究，衡量主流图像与视频生成模型在组合式文本到视觉生成各维度的表现。我们还将自动化评估指标与收集到的人类评分进行对比，发现VQAScore——一种衡量VQA（视觉问答）模型认为图像准确呈现提示的可能性的指标——显著优于CLIPScore等现有指标。此外，VQAScore可通过仅对少量（3至9张）候选图像排序，以黑盒方式（无需微调）提升生成效果。在提升DALL-E 3和Stable Diffusion的人类对齐评分方面，按VQAScore排序的效果是PickScore、HPSv2、ImageReward等其他评分方法的2至3倍，在需要高级视觉-语言推理的组合式提示上表现尤为突出。我们发布了全新的GenAI-Rank基准，包含超40000条人类评分，用于评估评分指标对同一提示生成图像的排序能力。最后，我们探讨了VQAScore的潜在改进方向，例如处理细粒度视觉细节。我们将发布全部超80000条人类评分，为生成模型与自动化指标的科学基准测试提供支持。

英文摘要：

While text-to-visual models now produce photo-realistic images and videos, they struggle with compositional text prompts involving attributes, relationships, and higher-order reasoning such as logic and comparison. In this work, we conduct an extensive human study on GenAI-Bench to evaluate the performance of leading image and video generation models in various aspects of compositional text-to-visual generation. We also compare automated evaluation metrics against our collected human ratings and find that VQAScore -- a metric measuring the likelihood that a VQA model views an image as accurately depicting the prompt -- significantly outperforms previous metrics such as CLIPScore. In addition, VQAScore can improve generation in a black-box manner (without finetuning) via simply ranking a few (3 to 9) candidate images. Ranking by VQAScore is 2x to 3x more effective than other scoring methods like PickScore, HPSv2, and ImageReward at improving human alignment ratings for DALL-E 3 and Stable Diffusion, especially on compositional prompts that require advanced visio-linguistic reasoning. We release a new GenAI-Rank benchmark with over 40,000 human ratings to evaluate scoring metrics on ranking images generated from the same prompt. Lastly, we discuss promising areas for improvement in VQAScore, such as addressing fine-grained visual details. We will release all human ratings (over 80,000) to facilitate scientific benchmarking of both generative models and automated metrics.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。