CREval: 一种自动化可解释的评估方法用于复杂指令下的创意图像操纵
CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructions
AI总结：
本文提出CREval，一种基于问答的自动化评估框架，用于评估复杂指令下的创意图像编辑任务，通过CREval-Bench基准测试揭示模型在复杂创意任务中的表现及挑战。
摘要：
基于指令的多模态图像操纵近年来取得了快速进展。然而，现有评估方法缺乏系统且与人类对齐的框架来评估模型在复杂和创意编辑任务上的性能。为解决这一差距，我们提出了CREval，一种完全自动化的问答（QA）基于评估流程，克服了多模态大语言模型（MLLMs）评分的不完整性和低可解释性。同时，我们引入了CREval-Bench，一个专门设计用于复杂指令下的创意图像操纵的综合基准。CREval-Bench涵盖三个类别和九个创意维度，包含超过800个编辑样本和13K个评估查询。利用此流程和基准，我们系统地评估了一组多样化的最新开源和闭源模型。结果表明，尽管闭源模型在复杂和创意任务上通常优于开源模型，但所有模型仍难以有效完成此类编辑。此外，用户研究显示CREval的自动化指标与人类判断之间有一致性。因此，CREval为评估复杂和创意图像操纵任务的图像编辑模型提供了可靠的依据，并突显了未来研究的关键挑战和机遇。
英文摘要：
Instruction-based multimodal image manipulation has recently made rapid progress. However, existing evaluation methods lack a systematic and human-aligned framework for assessing model performance on complex and creative editing tasks. To address this gap, we propose CREval, a fully automated question-answer (QA)-based evaluation pipeline that overcomes the incompleteness and poor interpretability of opaque Multimodal Large Language Models (MLLMs) scoring. Simultaneously, we introduce CREval-Bench, a comprehensive benchmark specifically designed for creative image manipulation under complex instructions. CREval-Bench covers three categories and nine creative dimensions, comprising over 800 editing samples and 13K evaluation queries. Leveraging this pipeline and benchmark, we systematically evaluate a diverse set of state-of-the-art open and closed-source models. The results reveal that while closed-source models generally outperform open-source ones on complex and creative tasks, all models still struggle to complete such edits effectively. In addition, user studies demonstrate strong consistency between CREval's automated metrics and human judgments. Therefore, CREval provides a reliable foundation for evaluating image editing models on complex and creative image manipulation tasks, and highlights key challenges and opportunities for future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
