AI能绘制科学吗？评估文本到图像和多模态模型生成科学图形的基准
Can AI Draw Science? A Benchmark for Evaluating Scientific Figure Generation by Text-to-Image and Multimodal Models
AI总结：
提出SciDraw-Bench基准，通过32个结构化任务和四维评估协议（文本保真度、语义正确性、结构质量、惯例遵循），评估文本到图像模型生成科学图形的能力。
摘要：
文本到图像和多模态生成模型越来越多地被用于生成科学图形，如机制图、实验设计示意图、概念框架和图形摘要。然而，现有的图像生成基准（如GenEval、T2I-CompBench、DPG-Bench）评估自然图像，并衡量组合性、对象计数或逼真度。它们都没有衡量生成的科学图形的可用性：正确且可读的文本标签、实体及其关系的忠实描绘、连贯的图解结构以及遵守学科绘图惯例。我们引入了SciDraw-Bench，这是一个包含32个结构化科学图形生成任务的基准，涵盖八种图形类型和十个学科，其中每个任务将一个自然语言提示与一个机器可检查的规范配对，该规范包含所需标签、关系、组件、惯例和负面约束。我们提出了一个四维评估协议：文本保真度（基于OCR的标签召回率和字符错误率）、语义正确性（视觉语言模型根据规范进行判断）、结构质量和惯例遵循，以及一个元评估协议和初步的评估者间可靠性分析（人工评分验证正在进行中）。我们评估了一个特定领域的系统SciDraw AI，与代表性的通用文本到图像模型进行比较，并概述了一个代码到图形的基线作为计划扩展。在对所有八种图形类型的试点中，特定领域系统在每个维度和图形类型上都显著优于通用基线，在语义正确性和惯例遵循方面差距最大；文本保真度仍然是所有系统最困难的维度。
英文摘要：
Text-to-image and multimodal generative models are increasingly used to produce scientific figures such as mechanism diagrams, experimental-design schematics, conceptual frameworks, and graphical abstracts. Yet existing image-generation benchmarks (e.g., GenEval, T2I-CompBench, DPG-Bench) evaluate natural images and measure compositionality, object counting, or photorealism. None of them measure what makes a generated scientific figure usable: correct and legible text labels, faithful depiction of entities and their relations, coherent diagrammatic structure, and adherence to disciplinary drawing conventions. We introduce SciDraw-Bench, a benchmark of 32 structured scientific-figure generation tasks spanning eight figure types and ten disciplines, where each task pairs a natural-language prompt with a machine-checkable specification of required labels, relations, components, conventions, and negative constraints. We propose a four-dimensional evaluation protocol: Text Fidelity (OCR-based label recall and character error rate), Semantic Correctness (vision-language-model judging against the specification), Structural Quality, and Convention Adherence, together with a meta-evaluation protocol and a preliminary inter-judge reliability analysis (human-rating validation is ongoing). We evaluate a domain-specific system, SciDraw AI, against representative general-purpose text-to-image models, and outline a code-to-figure baseline as a planned extension. In a pilot over all eight figure types, the domain-specific system substantially outperforms the general-purpose baselines on every dimension and figure type, with the largest gaps on semantic correctness and convention adherence; text fidelity remains the hardest dimension for all systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
