arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.14967cs.SEcs.AIcs.LG
收藏

MermaidSeqBench: 一种用于自然语言到Mermaid序列图生成的评估基准

MermaidSeqBench: An Evaluation Benchmark for NL-to-Mermaid Sequence Diagram Generation

IBM Research(IBM研究院)

Basel Shbita, Farhan Ahmed, Chad DeLuca

AI总结:

本文提出MermaidSeqBench,通过混合方法构建132个样本,评估LLM生成Mermaid序列图的能力,揭示模型间显著能力差异。

摘要:

大型语言模型(LLMs)在从自然语言描述生成结构化图示方面展现出巨大潜力,特别是Mermaid序列图在软件工程中的应用。然而,缺乏评估LLM在此任务上正确性的现有基准阻碍了这些模型在生产环境中的可靠部署。为解决这一不足,我们引入MermaidSeqBench,一个由人类验证和LLM合成扩展构成的评估基准,用于评估LLM从自然语言提示生成Mermaid序列图的能力。该基准包含132个样本,通过人类验证流程、LLM增强和规则扩展的混合方法开发。评估使用LLM作为裁判模型,评估语法正确性、激活处理、错误处理和实用性等细粒度指标。为了展示基准的有效性和灵活性,我们在多个最先进的LLM上进行了初始评估,使用多个LLM裁判揭示了模型和评估模式之间的显著能力差距。MermaidSeqBench为评估结构化图示生成提供了基础,并建立了现实世界软件工程部署所需的正确性标准。

英文摘要:

Large language models (LLMs) have shown great promise in generating structured diagrams from natural language descriptions, particularly Mermaid sequence diagrams for software engineering. However, the lack of existing benchmarks to assess the LLM's correctness on this task hinders rigorous, systematic evaluation and principled comparison of model capabilities on this task. To address this shortcoming, we introduce MermaidSeqBench, a human-verified and synthetically extended benchmark for assessing LLM capabilities in generating Mermaid sequence diagrams from natural language prompts. The benchmark consists of 132 samples developed via a hybrid methodology of human-verified flows, LLM-based augmentation, and rule-based expansion. The evaluation uses an LLM-as-a-judge model to assess generation across various fine-grained metrics such as syntax correctness, activation handling, error handling, and practical usability. To demonstrate the effectiveness and flexibility of our benchmark, we perform initial evaluations on numerous state-of-the-art LLMs with multiple LLM judges which reveal significant capability gaps across models and evaluation modes. MermaidSeqBench provides a foundation for evaluating structured diagram generation and advances the scientific understanding of LLM capabilities and limitations in structured generation tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑