arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.02742cs.SEcs.AIcs.CL
收藏

DSL代码生成的比较研究:微调与优化检索增强

A Comparative Study of DSL Code Generation: Fine-Tuning vs. Optimized Retrieval Augmentation

Microsoft Corporation(微软公司)

Nastaran Bassamzadeh, Chhaya Methani

AI总结:

本文比较了微调与优化检索增强生成在DSL代码生成中的效果,发现优化后的RAG模型在相似度上可媲美微调模型,并对新API更具优势。

摘要:

近年来,随着大型语言模型(LLMs)的出现,自然语言到代码生成取得了显著进展。虽然针对C、C++和Python等通用语言的生成能力已大幅提升,但LLMs在处理领域特定语言(DSLs)中的自定义函数名时仍然面临困难。这会导致更高的幻觉率和语法错误,尤其是对于包含大量自定义函数名的DSL而言。此外,函数名的持续更新进一步增加了挑战,因为LLMs需要保持最新状态。在本文中,我们提出了在使用LLMs进行DSL生成时对检索增强生成(RAG)的优化方案,并进行了比较这些策略的消融研究。我们使用一种DSL生成了训练和测试数据集,以表示公共领域中约700个API的自动化任务。我们使用训练数据集针对该DSL微调了一个Codex模型。我们的结果表明,微调模型在代码相似度指标上得分最高。通过我们的RAG优化,我们在相似度指标上达到了同等水平。然而,编译率显示两种模型仍然多次出现语法错误,基于RAG的方法高出2个百分点。相反,RAG模型在API名称上的幻觉率落后1个百分点,在API参数键上落后2个百分点。我们得出结论,优化后的RAG模型可以匹配微调模型的质量,并为新的、未见过的API提供优势。

英文摘要:

Natural Language to Code Generation has made significant progress in recent years with the advent of Large Language Models(LLMs). While generation for general-purpose languages like C, C++, and Python has improved significantly, LLMs struggle with custom function names in Domain Specific Languages or DSLs. This leads to higher hallucination rates and syntax errors, specially for DSLs having a high number of custom function names. Additionally, constant updates to function names add to the challenge as LLMs need to stay up-to-date. In this paper, we present optimizations for using Retrieval Augmented Generation (or RAG) with LLMs for DSL generation along with an ablation study comparing these strategies. We generated a train as well as test dataset with a DSL to represent automation tasks across roughly 700 APIs in public domain. We used the training dataset to fine-tune a Codex model for this DSL. Our results showed that the fine-tuned model scored the best on code similarity metric. With our RAG optimizations, we achieved parity for similarity metric. The compilation rate, however, showed that both the models still got the syntax wrong many times, with RAG-based method being 2 pts better. Conversely, hallucination rate for RAG model lagged by 1 pt for API names and by 2 pts for API parameter keys. We conclude that an optimized RAG model can match the quality of fine-tuned models and offer advantages for new, unseen APIs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑