寻找主观真相:收集200万张选票以全面评估生成式AI模型
Finding the Subjective Truth: Collecting 2 Million Votes for Comprehensive Gen-AI Model Evaluation
AI总结:
针对文本到图像模型主观评估难的问题,提出高效标注框架,收集超200万条标注,全面评估并排名四个知名模型,证明多元化全球标注员可显著降低偏见风险。
摘要:
高效评估文本到图像模型的性能十分困难,因为这本质上需要主观判断和人类偏好,使得比较不同模型和量化最先进水平变得困难。借助Rapidata的技术,我们提出了一个高效的标注框架,从多元化、全球化的标注员池中收集人类反馈。我们的研究在4,512张图像上收集了超过200万条标注,评估了四个知名模型(DALL-E 3、Flux.1、MidJourney和Stable Diffusion)在风格偏好、连贯性和文本到图像对齐方面的表现。我们证明,我们的方法使得基于庞大标注员池全面排名图像生成模型成为可能,并表明多元化的标注员人口统计学特征反映了世界人口分布,显著降低了偏见风险。
英文摘要:
Efficiently evaluating the performance of text-to-image models is difficult as it inherently requires subjective judgment and human preference, making it hard to compare different models and quantify the state of the art. Leveraging Rapidata's technology, we present an efficient annotation framework that sources human feedback from a diverse, global pool of annotators. Our study collected over 2 million annotations across 4,512 images, evaluating four prominent models (DALL-E 3, Flux.1, MidJourney, and Stable Diffusion) on style preference, coherence, and text-to-image alignment. We demonstrate that our approach makes it feasible to comprehensively rank image generation models based on a vast pool of annotators and show that the diverse annotator demographics reflect the world population, significantly decreasing the risk of biases.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。