为什么大语言模型在OCL生成中失败？一种图推理视角

中文摘要 AI 辅助

大语言模型（LLMs）越来越多地被用于从自然语言规范和UML类图生成对象约束语言（OCL）约束。然而，现有工作主要侧重于提高准确性，对这些模型为何失败的理解有限。目标：本研究调查LLM在OCL生成中失败的潜在原因，将任务框定为对UML类图的图推理问题。方法：我们使用PathOCL数据集对六个最先进的大语言模型进行了实证评估。我们分析了UML结构属性（例如，导航深度和模型复杂度）、词汇相似性、提示排序策略以及图感知提示对OCL正确性的影响。结果：我们发现，随着导航深度和结构复杂性的增加，OCL生成性能显著下降。词汇相似性影响有限，而UML元素的文本排序影响性能。基于图的提示产生了部分改进，但并未消除结构推理错误。结论：OCL生成主要受图推理限制的约束，而非纯粹的语言因素。这些结果突显了结构推理是当前LLM在模型驱动工程任务中的关键瓶颈。

英文摘要

Large Language Models (LLMs) are increasingly used to generate Object Constraint Language (OCL) constraints from natural language specifications and UML class diagrams. However, existing work mainly focuses on improving accuracy, with limited understanding of why these models fail. Aims. This study investigates the underlying causes of LLM failures in OCL generation, framing the task as a graph reasoning problem over UML class diagrams. Method. We conduct an empirical evaluation using the PathOCL dataset across six state-of-the-art LLMs. We analyze the impact of UML structural properties (e.g., navigation depth and model complexity), lexical similarity, prompt ordering strategies, and graph-aware prompting on OCL correctness. Results. We find that OCL generation performance significantly degrades with increasing navigation depth and structural complexity. Lexical similarity has limited influence, while textual ordering of UML elements affects performance. Graph-based prompting yields partial improvements but does not eliminate structural reasoning errors. Conclusions. OCL generation is primarily constrained by graph reasoning limitations rather than purely linguistic factors. These results highlight structural reasoning as a key bottleneck for current LLMs in model-driven engineering tasks.

发表机构

• Polytechnique Montréal（蒙特利尔高等理工学院）
• Concordia University（康考迪亚大学）
• Trent University（特伦特大学）

机构由 AI 辅助整理，请以论文原文为准。