arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.16244cs.SEcs.AI
收藏

法典提示工程生成OCL的实证研究

On Codex Prompt Engineering for OCL Generation: An Empirical Study

Polytechnique Montréal(蒙特利尔综合理工学院)

Seif Abukhalaf, Mohammad Hamdaqa, Foutse Khomh

摘要:

对象约束语言(OCL)是一种声明性语言,它向MOF模型添加约束和对象查询表达式。尽管OCL有潜力为UML模型提供精确性和简洁性,但OCL不熟悉的语法阻碍了它的采用。LLM的最新进展,如GPT-3,已经在许多NLP任务中显示了它们的能力,包括语义解析和文本生成。Codex是GPT-3的后代,已经在GitHub的公开代码上进行了微调,可以用许多程序设计语言生成代码。我们调查的可靠性OCL约束产生的Codex从自然语言规范。为了实现这一点,我们编译了一个包含15个UML模型和168个规范的数据集,并制作了一个提示模板,其中包含用于填充UML信息和目标任务的插槽,使用零次学习和少样本学习方法。通过测量生成的OCL约束的语法有效性和执行准确性度量,我们发现,丰富的提示与UML信息和启用少样本学习增加了生成的OCL约束的可靠性。此外,结果揭示了基于句子嵌入之间的生成的OCL约束和人类书面的地面真理,这意味着一个层次的清晰度和可理解性,在生成的OCL约束的Codex。

英文摘要:

The Object Constraint Language (OCL) is a declarative language that adds constraints and object query expressions to MOF models. Despite its potential to provide precision and conciseness to UML models, the unfamiliar syntax of OCL has hindered its adoption. Recent advancements in LLMs, such as GPT-3, have shown their capability in many NLP tasks, including semantic parsing and text generation. Codex, a GPT-3 descendant, has been fine-tuned on publicly available code from GitHub and can generate code in many programming languages. We investigate the reliability of OCL constraints generated by Codex from natural language specifications. To achieve this, we compiled a dataset of 15 UML models and 168 specifications and crafted a prompt template with slots to populate with UML information and the target task, using both zero- and few-shot learning methods. By measuring the syntactic validity and execution accuracy metrics of the generated OCL constraints, we found that enriching the prompts with UML information and enabling few-shot learning increases the reliability of the generated OCL constraints. Furthermore, the results reveal a close similarity based on sentence embedding between the generated OCL constraints and the human-written ones in the ground truth, implying a level of clarity and understandability in the generated OCL constraints by Codex.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑