Knossos 与 Ariadne：使用视觉语言模型对完整图表拓扑提取进行基准测试与学习

中文摘要 AI 辅助

结构图被广泛用于表示科学、工程、程序和空间领域中的复杂系统和关系信息。近年来，视觉语言模型（VLMs）在识别图表元素和推理其内容方面能力日益增强，而完整的图表拓扑提取仍相对未被充分探索。本文研究从图表到图的拓扑提取任务：提取所有图表实体及其之间的完整关系。为支持该任务的大规模监督训练和系统评估，我们引入了 Knossos，一个包含六个不同领域、共 19,200 个图表的基准，拥有 245,179 个节点和 439,740 条边。其符号化生成过程提供了渲染图表与完整拓扑、关系类型和连接器几何注释之间的精确对齐。为解决完整拓扑提取的建模挑战，我们还提出了 Ariadne，一个结构化框架，将任务分解为节点清单提取和基于源的条件边预测。大量实验表明，在 Knossos 上训练能显著提升较小开源视觉语言模型的完整拓扑提取能力。Ariadne 在匹配监督下进一步优于一步提取方法，展示了结构化分解的额外优势。它在 Knossos 上评估的方法中取得了最高的平均边 F1 分数，同时其两种骨干变体在真实世界外部基准上也优于未适配的对应模型。代码和基准可在 https:// 此 URL 获取。

英文摘要

Structural diagrams are widely used to represent complex systems and relational information across scientific, engineering, procedural, and spatial domains. Recent vision-language models (VLMs) have become increasingly capable of recognizing diagram elements and reasoning about their content, while complete diagram topology extraction remains comparatively underexplored. In this paper, we study diagram-to-graph topology extraction: extracting all diagram entities and the complete relations among them. To enable large-scale supervised training and systematic evaluation of this task, we introduce Knossos, a benchmark of 19,200 diagrams across six diverse domains, with 245,179 nodes and 439,740 edges. Its symbolic generation process provides exact alignment between rendered diagrams and annotations of complete topology, relation types, and connector geometry. To address the modeling challenge of complete topology extraction, we also present Ariadne, a structured framework that decomposes the task into node inventory extraction and source-conditioned edge prediction. Extensive experiments show that training on Knossos substantially improves complete topology extraction in smaller open-source VLMs. Ariadne further improves over one-step extraction under matched supervision, demonstrating the additional benefit of structured decomposition. It achieves the highest average Edge F1 among the evaluated methods on Knossos, while both backbone variants also improve over their unadapted counterparts on the real-world external benchmark. Code and benchmark are available at https://github.com/bangwayne/knossos_Ariadne_Public.

发表机构

• Rutgers University（罗格斯大学）
• Meta
• NEC Labs America（NEC美国实验室）

机构由 AI 辅助整理，请以论文原文为准。