PaintBench: 精确视觉编辑的确定性评估
PaintBench: Deterministic Evaluation of Precise Visual Editing
AI总结：
提出PaintBench基准，通过程序化生成20种基本视觉编辑操作，实现确定性像素级评估，发现当前模型性能低（最高mIoU 17.1%），并揭示任务分解和场景变化的影响。
摘要：
虽然当前的多模态模型在开放式视觉编辑方面表现熟练，但执行精确的单答案编辑仍然是一个重要障碍。为了探究这一挑战，我们引入了PaintBench，一个动态可扩展的基准测试，针对四个类别的20种基本精确视觉编辑操作：几何变换、结构操作、颜色变化和符号推理。具有可配置复杂性的程序化生成实现了有效无限、抗污染的评估套件，而确定性像素级评估消除了对易偏见的评判模型的依赖。在11个图像编辑模型中，我们发现整体性能较低，当前表现最佳的行业领先者仅得17.1%（mIoU）。任务分解揭示了特别具有挑战性的操作类型（几何变换、大多数结构操作、基于公式的颜色变化）和模型特定的专长。细粒度的基准诊断进一步显示了由对象数量、背景复杂性、配色方案和编辑区域大小等场景变化引起的性能下降。为了测试PaintBench分数对应用任务性能的泛化能力，我们创建了一个用于数据可视化编辑的程序化确定性评估（TinyGrafixBench），并发现其与PaintBench分数之间存在强线性相关性（$R^2 = 0.91$, $p < 0.001$）。总之，PaintBench为衡量和推动精确多模态视觉编辑的进展提供了严格的基础。
英文摘要：
While current multimodal models are proficient at open-ended visual editing, executing precise single-answer edits remains an important obstacle. To probe this challenge, we introduce PaintBench, a dynamically scalable benchmark targeting 20 fundamental precise visual editing operations across four categories: geometric transformation, structural manipulation, color change, and symbolic reasoning. Procedural generation with configurable complexity enables an effectively infinite, contamination-resistant evaluation suite, and deterministic pixel-level evaluation eliminates reliance on bias-prone judge models. Across 11 image editing models, we find overall low performance, with the current highest-performing industry leader scoring only 17.1% (mIoU). Task decomposition reveals especially challenging operation types (geometric transformation, most structural manipulation, formula-based color change) and model-specific specializations. Fine-grained benchmark diagnostics further show performance degradations induced by scene variations in object count, background complexity, color scheme, and edit-region size. To test generalization of PaintBench scores to applied task performance, we create a procedural, deterministic evaluation for data visualization editing (TinyGrafixBench) and find strong linear correlation with PaintBench scores ($R^2 = 0.91$, $p < 0.001$). Altogether, PaintBench provides a rigorous foundation for measuring and driving progress in precise multimodal visual editing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
