GenAI Arena：一个面向生成式模型的开放评估平台
GenAI Arena: An Open Evaluation Platform for Generative Models
AI总结：
本文提出开放评估平台GenAI-Arena，通过用户投票评估图像和视频生成模型，并发布偏好数据集GenAI-Bench，发现现有GPT-4o等多模态模型在评估视觉内容时准确率仅49.19%。
摘要：
生成式AI在图像和视频生成等领域取得了显著进展。这些进步由创新的算法、架构和数据驱动。然而，生成式模型的快速激增凸显了一个关键缺口：缺乏可靠的评估指标。当前的自动评估（如FID、CLIP、FVD等）通常无法捕捉生成输出所关联的细微质量和用户满意度。本文提出了一个开放平台GenAI-Arena来评估不同的图像和视频生成模型，用户可以积极参与评估这些模型。通过利用集体用户反馈和投票，GenAI-Arena旨在提供一种更民主、更准确的模型性能衡量标准。它涵盖了文本到图像生成、文本到视频生成和图像编辑三个任务。目前，我们共涵盖了35个开源生成模型。GenAI-Arena已运行七个月，累计获得社区超过9000次投票。我们描述了该平台，分析了数据，并解释了用于对模型进行排名的统计方法。为了进一步推动基于模型的评估指标的研究，我们发布了这三个任务的偏好数据的清理版本，即GenAI-Bench。我们提示现有的多模态模型（如Gemini和GPT-4o）来模仿人类投票。我们通过将模型投票与人类投票进行比较来计算准确率，以了解它们的评判能力。结果显示，现有的多模态模型在评估生成的视觉内容方面仍然落后，即使是最好的模型GPT-4o在三个生成任务上的平均准确率也仅为49.19。由于在复杂视觉场景中缺乏指令遵循和推理能力，开源MLLMs的表现甚至更差。
英文摘要：
Generative AI has made remarkable strides to revolutionize fields such as image and video generation. These advancements are driven by innovative algorithms, architecture, and data. However, the rapid proliferation of generative models has highlighted a critical gap: the absence of trustworthy evaluation metrics. Current automatic assessments such as FID, CLIP, FVD, etc often fail to capture the nuanced quality and user satisfaction associated with generative outputs. This paper proposes an open platform GenAI-Arena to evaluate different image and video generative models, where users can actively participate in evaluating these models. By leveraging collective user feedback and votes, GenAI-Arena aims to provide a more democratic and accurate measure of model performance. It covers three tasks of text-to-image generation, text-to-video generation, and image editing respectively. Currently, we cover a total of 35 open-source generative models. GenAI-Arena has been operating for seven months, amassing over 9000 votes from the community. We describe our platform, analyze the data, and explain the statistical methods for ranking the models. To further promote the research in building model-based evaluation metrics, we release a cleaned version of our preference data for the three tasks, namely GenAI-Bench. We prompt the existing multi-modal models like Gemini, and GPT-4o to mimic human voting. We compute the accuracy by comparing the model voting with the human voting to understand their judging abilities. Our results show existing multimodal models are still lagging in assessing the generated visual content, even the best model GPT-4o only achieves an average accuracy of 49.19 across the three generative tasks. Open-source MLLMs perform even worse due to the lack of instruction-following and reasoning ability in complex vision scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
