arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.06013cs.CVcs.AI
收藏

GenArena: 如何在视觉生成任务中实现人对齐的评估?

GenArena: How Can We Achieve Human-Aligned Evaluation for Visual Generation Tasks?

University of Science(科学大学);Shanghai Innovation Institute(上海创新研究所);Tencent(腾讯);National University of Singapore(新加坡国立大学)

Ruihang Li, Leigang Qu, Jingxu Zhang, Dongnan Gui, Mengde Xu, Xiaosong Zhang, Han Hu, Wenjie Wang, Jiaqi Wang

AI总结:

GenArena通过成对比较范式实现视觉生成任务的人对齐评估,使开源模型超越专有模型,提升评估准确性达20%以上。

摘要:

视觉生成模型的快速发展已经超过了传统评估方法,需要采用视觉-语言模型作为替代裁判。在本工作中,我们系统地研究了现行的绝对点wise评分标准在广泛视觉生成任务中的可靠性。我们的分析揭示了这种范式由于随机不一致性和与人类感知的不匹配而受到限制。为了解决这些限制,我们引入了GenArena,一个统一的评估框架,利用成对比较范式来确保稳定且人对齐的评估。关键地,我们的实验揭示了一个变革性的发现,即仅仅采用这种成对协议可以使现成的开源模型超越顶级专有模型。值得注意的是,我们的方法将评估准确性提高了超过20%,并实现了与权威LMArena排行榜的斯皮尔曼相关性为0.86,远远超过了点wise方法的0.36相关性。基于GenArena,我们对最先进的视觉生成模型在多样化任务中进行基准测试,为社区提供了严格且自动化的视觉生成评估标准。

英文摘要:

The rapid advancement of visual generation models has outpaced traditional evaluation approaches, necessitating the adoption of Vision-Language Models as surrogate judges. In this work, we systematically investigate the reliability of the prevailing absolute pointwise scoring standard, across a wide spectrum of visual generation tasks. Our analysis reveals that this paradigm is limited due to stochastic inconsistency and poor alignment with human perception. To resolve these limitations, we introduce GenArena, a unified evaluation framework that leverages a pairwise comparison paradigm to ensure stable and human-aligned evaluation. Crucially, our experiments uncover a transformative finding that simply adopting this pairwise protocol enables off-the-shelf open-source models to outperform top-tier proprietary models. Notably, our method boosts evaluation accuracy by over 20% and achieves a Spearman correlation of 0.86 with the authoritative LMArena leaderboard, drastically surpassing the 0.36 correlation of pointwise methods. Based on GenArena, we benchmark state-of-the-art visual generation models across diverse tasks, providing the community with a rigorous and automated evaluation standard for visual generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑