On Codex Prompt Engineering for OCL Generation: An Empirical Study
Polytechnique Montréal(蒙特利尔综合理工学院)
Seif Abukhalaf, Mohammad Hamdaqa, Foutse Khomh
备注:
10 pages. Full abstract in the pre-print. Accepted to be published to the 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR)
The Object Constraint Language (OCL) is a declarative language that adds constraints and object query expressions to MOF models. Despite its potential to provide precision and conciseness to UML models, the unfamiliar syntax of OCL has hindered its adoption. Recent advancements in LLMs, such as GPT-3, have shown their capability in many NLP tasks, including semantic parsing and text generation. Codex, a GPT-3 descendant, has been fine-tuned on publicly available code from GitHub and can generate code in many programming languages. We investigate the reliability of OCL constraints generated by Codex from natural language specifications. To achieve this, we compiled a dataset of 15 UML models and 168 specifications and crafted a prompt template with slots to populate with UML information and the target task, using both zero- and few-shot learning methods. By measuring the syntactic validity and execution accuracy metrics of the generated OCL constraints, we found that enriching the prompts with UML information and enabling few-shot learning increases the reliability of the generated OCL constraints. Furthermore, the results reveal a close similarity based on sentence embedding between the generated OCL constraints and the human-written ones in the ground truth, implying a level of clarity and understandability in the generated OCL constraints by Codex.
LEGO: An LLM Skill-Based Front-End Design Generation Platform
School of IC, Peking University(北京大学集成电路学院);School of EECS, Peking University(北京大学电子信息技术学院);School of Microelectronics, Xidian University(西安电子科技大学微电子学院);Institute of EDA, Peking University(北京大学EDA研究院);Beijing Advanced Innovation Center for IC(北京集成电路先进创新中心)
Querying an astronomical database using large language models: the ALeRCE text-to-SQL system
Department of Electrical Engineering, University of Chile, Av. Tupper 2007, Santiago, Chile Millennium Institute of Astrophysics (MAS), Nuncio Monseñor Sótero Sanz 100, Providencia, Santiago, Chile Data;Artificial Intelligence Initiative (ID\&IA), Universidad de Chile Center for Mathematical Modeling, Universidad de Chile, Beauchef 851, North building, 7th floor, Santiago 8320000, Chile Departamento de Astronom\'ia, Universidad de Chile, Casilla 36D, Santiago, Chile Department of Computer Science, Universidad de Concepción, Edmundo Larenas 219, Concepción, Chile Center for Data;Artificial Intelligence, Universidad de Concepción, Edmundo Larenas 310, Concepción, Chile Heidelberg Institute for Theoretical Studies, Heidelberg, Baden-Württemberg, Germany Instituto de Alta Investigación, Universidad de Tarapacá, Casilla 7D, Arica, 1010000, Chile European Southern Observatory, Karl-Schwarzschild-Strasse 2, 85748 Garching bei München, Germany Instituto de Astrofísica, Facultad de Física, Pontificia Universidad Católica de Chile, Casilla 306, Santiago 22, Chile Centro de Astroingeniería, Pontificia Universidad Católica de Chile, Av. Vicuña Mackenna 4860, 7820436 Macul, Santiago, Chile Instituto de Estudios Astrof\'isicos, Facultad de Ingenier\'ia y Ciencias, Universidad Diego Portales, Av. Ej\'ercito Libertador 441, Santiago, Chile Centro Interdisciplinario de Data Science, Facultad de Ingenier\'ia y Ciencias, Universidad Diego Portales, Av. Ej\'ercito Libertador 441, Santiago, Chile
提出基于大语言模型的文本到SQL系统,通过上下文学习和逐步生成框架(模式链接、查询分类、提示分解、自纠正)实现自然语言查询天文数据库,在ALeRCE数据集上评估13个模型,Claude Opus 4.6等表现最佳。