用于流程图图像到代码生成的在线无参考评估框架
An Online Reference-Free Evaluation Framework for Flowchart Image-to-Code Generation
AI总结:
针对流程图图像到代码生成在生产中无真实代码难以评估输出质量的问题,提出无参考评估框架,通过RecallOCR和PrecisionVE两个指标及调和均值F1OCR-VE监测生成质量,在FlowVQA数据集验证了其可靠性。
摘要:
视觉语言模型(VLMs)越来越多地用于文档处理管道,将流程图图像转换为结构化代码(如Mermaid)。在生产中,这些系统处理不存在真实代码的任意输入,难以评估输出质量。我们提出了一个无参考评估框架,在推理时仅使用输入图像和生成的输出监测流程图图像到代码的生成质量。该框架引入了两个自动指标:RecallOCR,通过OCR从输入图像中提取文本作为代理参考来估计内容覆盖率;PrecisionVE,通过视觉蕴含检测与原始图像相比的幻觉元素。它们的调和均值F1OCR-VE提供统一的质量分数。在FlowVQA数据集上的验证表明与真实指标高度一致,证实了该框架作为生产环境中连续质量监测的实用无参考替代方案的可靠性。
英文摘要:
Vision-Language Models (VLMs) are increasingly used in document processing pipelines to convert flowchart images into structured code (e.g., Mermaid). In production, these systems process arbitrary inputs for which no ground-truth code exists, making output quality difficult to assess. We propose a reference-free evaluation framework that monitors flowchart image-to-code generation quality at inference time, using only the input image and the generated output. The framework introduces two automated metrics: $\text{Recall}{\text{OCR}}$, which estimates content coverage by extracting text from the input image via OCR as a proxy reference, and $\text{Precision}{\text{VE}}$, which detects hallucinated elements through Visual Entailment against the original image. Their harmonic mean, $\text{F1}{\text{OCR-VE}}$, provides a unified quality score. Validation on the FlowVQA dataset shows strong agreement with ground-truth metrics (average Pearson's $r = 0.97$, $0.91$, and $0.94$ for Recall, Precision, and F1, respectively), confirming the framework's reliability as a practical, reference-free alternative for continuous quality monitoring in production settings.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。