AIBench：评估学术示意图生成中的视觉-逻辑一致性
AIBench: Evaluating Visual-Logical Consistency in Academic Illustration Generation
AI总结：
本文提出AIBench，首个通过VQA评估学术示意图逻辑正确性及VLM评估美观性的基准，通过四层问题验证生成示意图与论文的一致性，发现模型在该任务上的性能差异显著，且逻辑与美观难以同时优化。
摘要：
尽管图像生成通过快速进化提升了各种应用，但目前最先进的模型是否能生成适用于论文的学术示意图仍鲜有探索。直接比较或评估示意图与VLM存在固有缺陷，需依赖oracle多模态理解能力，难以处理长复杂文本和示意图。为此，我们提出AIBench，首个利用VQA评估学术示意图逻辑正确性及VLM评估美观性的基准。具体而言，我们从论文方法部分总结出逻辑图，设计四层问题，查询生成的示意图在不同尺度上是否与论文一致。我们的VQA方法在视觉-逻辑一致性评估上更准确详细，且依赖较少的判官VLM能力。通过高质量的AIBench，我们进行了广泛实验，得出模型在该任务上的性能差距显著大于通用模型，反映其复杂的推理和高密度生成能力。进一步，逻辑与美观难以同时优化，如手工制作的示意图。此外，实验还表明在测试时对两者能力的扩展显著提升了该任务的性能。
英文摘要：
Although image generation has boosted various applications via its rapid evolution, whether the state-of-the-art models are able to produce ready-to-use academic illustrations for papers is still largely unexplored. Directly comparing or evaluating the illustration with VLM is native but requires oracle multi-modal understanding ability, which is unreliable for long and complex texts and illustrations. To address this, we propose AIBench, the first benchmark using VQA for evaluating logic correctness of the academic illustrations and VLMs for assessing aesthetics. In detail, we designed four levels of questions proposed from a logic diagram summarized from the method part of the paper, which query whether the generated illustration aligns with the paper on different scales. Our VQA-based approach raises more accurate and detailed evaluations on visual-logical consistency while relying less on the ability of the judger VLM. With our high-quality AIBench, we conduct extensive experiments and conclude that the performance gap between models on this task is significantly larger than general ones, reflecting their various complex reasoning and high-density generation ability. Further, the logic and aesthetics are hard to optimize simultaneously as in handcrafted illustrations. Additional experiments further state that test-time scaling on both abilities significantly boosts the performance on this task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
