arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.19619cs.AIcs.SE
收藏

通过检索增强生成利用少样本学习增强语言模型中的代码翻译

Enhancing Code Translation in Language Models with Few-Shot Learning via Retrieval-Augmented Generation

Los Alamos National Laboratory(洛斯阿拉莫斯国家实验室)

Manish Bhattarai, Javier E. Santos, Shawn Jones, Ayan Biswas, Boian Alexandrov, Daniel O'Malley

AI总结:

本文提出基于检索增强生成(RAG)的少样本学习方法,通过动态检索代码翻译示例增强大型语言模型的翻译能力,在Fortran与CPP等语言间翻译任务中显著优于传统零样本方法。

摘要:

大型语言模型(LLMs)的出现显著推进了代码翻译领域,实现了编程语言之间的自动化翻译。然而,由于上下文理解不足,这些模型在处理复杂翻译任务时常常遇到困难。本文提出了一种新颖的方法,通过少样本学习(Few-Shot Learning)增强代码翻译,并结合检索技术。通过利用现有代码翻译的存储库,我们动态检索最相关的示例,以指导模型翻译新的代码片段。我们的方法基于检索增强生成(RAG),通过提供上下文示例供模型实时学习,显著提升了翻译质量。我们选择RAG而非传统微调方法,是因为它能够利用现有代码库或本地存储的代码语料库,从而无需大量重新训练即可动态适应多样化的翻译任务。我们在多样化数据集上进行了广泛实验,使用了开源LLM模型如Starcoder、Llama3-70B Instruct、CodeLlama-34B Instruct、Granite-34B Code Instruct和Mixtral-8x22B,以及商业LLM模型如GPT-3.5 Turbo和GPT-4o,结果表明我们的方法优于传统零样本方法,尤其是在Fortran和CPP之间的翻译。我们还探索了不同数量的shots,即在推理期间提供的示例数量,具体为1、2和3 shots,以及用于RAG的不同嵌入模型,包括Nomic-Embed、Starencoder和CodeBERT,以评估我们方法的稳健性和有效性。

英文摘要:

The advent of large language models (LLMs) has significantly advanced the field of code translation, enabling automated translation between programming languages. However, these models often struggle with complex translation tasks due to inadequate contextual understanding. This paper introduces a novel approach that enhances code translation through Few-Shot Learning, augmented with retrieval-based techniques. By leveraging a repository of existing code translations, we dynamically retrieve the most relevant examples to guide the model in translating new code segments. Our method, based on Retrieval-Augmented Generation (RAG), substantially improves translation quality by providing contextual examples from which the model can learn in real-time. We selected RAG over traditional fine-tuning methods due to its ability to utilize existing codebases or a locally stored corpus of code, which allows for dynamic adaptation to diverse translation tasks without extensive retraining. Extensive experiments on diverse datasets with open LLM models such as Starcoder, Llama3-70B Instruct, CodeLlama-34B Instruct, Granite-34B Code Instruct, and Mixtral-8x22B, as well as commercial LLM models like GPT-3.5 Turbo and GPT-4o, demonstrate our approach's superiority over traditional zero-shot methods, especially in translating between Fortran and CPP. We also explored varying numbers of shots i.e. examples provided during inference, specifically 1, 2, and 3 shots and different embedding models for RAG, including Nomic-Embed, Starencoder, and CodeBERT, to assess the robustness and effectiveness of our approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑