AtelierEval: 人类与LLM作为文本到图像提示器的代理评估
AtelierEval: Agentic Evaluation of Humans & LLMs as Text-to-Image Prompters
AI总结：
本文提出AtelierEval，首个统一基准，通过360个专家设计的任务量化提示能力，引入技能基于的记忆增强代理评估器，实现与人类专家的高相关性，验证了提示器在图像增强方向的优越性。
摘要：
文本到图像（T2I）系统日益依赖上游提示器，无论是人类还是多模态大语言模型（MLLMs），将用户意图转化为详细提示。然而，当前基准固定提示并仅评估T2I模型，忽略了上游组件的提示能力。我们引入AtelierEval，首个统一基准，通过360个专家设计的任务量化提示能力。基于认知观点，它涵盖三个任务类别，并使用现实挑战的分类学来实例化任务，为人类和MLLMs提供双接口。为了实现可扩展和可靠的评估，我们提出了AtelierJudge，一个技能基于、记忆增强的代理评估器。它为提示-图像对生成主观和客观评分，与人类专家的Spearman相关性达到0.79，接近人类表现。广泛实验在4个T2I后端上基准8个MLLMs和48个人类用户，验证AtelierEval作为稳健诊断工具的有效性，并揭示模仿优于规划，倡导未来提示器的图像增强方向。我们的工作已发布以支持未来研究。
英文摘要：
Text-to-image (T2I) systems increasingly rely on upstream prompters, either humans or multimodal large language models (MLLMs), to translate user intent into detailed prompts. Yet current benchmarks fix the prompt and only evaluate T2I models, leaving the prompting proficiency of this upstream component entirely unmeasured. We introduce AtelierEval, the first unified benchmark that quantifies prompting proficiency across 360 expert-crafted tasks. Grounded in a cognitive view, it spans three task categories and instantiates tasks using a taxonomy of real-world challenges, with a dual interface for both humans and MLLMs. To enable scalable and reliable evaluation, we propose AtelierJudge, a skill-based, memory-augmented agentic evaluator. It produces subjective and objective scores for prompt-image pairs, achieving a Spearman correlation of 0.79 with human experts, approaching human performance. Extensive experiments benchmark 8 MLLMs against 48 human users across 4 T2I backends, validate AtelierEval as a robust diagnostic tool, and reveal the superiority of mimicry over planning, advocating for an image-augmented direction for future prompters. Our work is released to support future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
