HYPE-EDIT-1：衡量前沿图像编辑模型可靠性的基准
HYPE-EDIT-1: Benchmark for Measuring Reliability in Frontier Image Editing Models
AI总结：
HYPE-EDIT-1是一个评估前沿图像编辑模型可靠性的基准，通过100个任务测试模型通过率和有效成本，揭示了低单价模型在考虑重试和人工审查后的高总成本。
摘要：
图像编辑模型的公开演示通常是最佳样本；真实工作流程需要重试和审查时间。我们引入HYPE-EDIT-1，这是一个包含100个任务的参考-based营销/设计编辑基准，采用二元通过/失败判断。对于每个任务，我们生成10个独立输出以估算每次尝试通过率、通过@10、在重试限制下的预期尝试次数，以及结合模型价格和人工审查时间的有效成功编辑成本。我们发布了50个公开任务，并维护一个50个任务的隐藏私人分割用于服务器端评估，加上为VLM和基于人类判断的标准JSON方案和工具。在评估的模型中，每次尝试通过率从34%到83%不等，有效成功成本从0.66美元到1.42美元不等。价格较低的模型在考虑重试和人工审查的总有效成本时更昂贵。
英文摘要：
Public demos of image editing models are typically best-case samples; real workflows pay for retries and review time. We introduce HYPE-EDIT-1, a 100-task benchmark of reference-based marketing/design edits with binary pass/fail judging. For each task we generate 10 independent outputs to estimate per-attempt pass rate, pass@10, expected attempts under a retry cap, and an effective cost per successful edit that combines model price with human review time. We release 50 public tasks and maintain a 50-task held-out private split for server-side evaluation, plus a standardized JSON schema and tooling for VLM and human-based judging. Across the evaluated models, per-attempt pass rates span 34-83 percent and effective cost per success spans USD 0.66-1.42. Models that have low per-image pricing are more expensive when you consider the total effective cost of retries and human reviews.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
