arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.08553cs.SE
收藏

通过传授大语言模型经验提升预训练语言模型的能力

Improving the Ability of Pre-trained Language Model by Imparting Large Language Model's Experience

Xin Yin, Chao Ni, Xiaodan Xu, Xinrui Li, Xiaohu Yang

AI总结:

针对大模型少样本下非生成类软件工程任务表现差、小模型微调依赖高质量数据的问题,提出用大语言模型生成领域数据增强预训练语言模型的方法,在故障定位等任务上取得显著性能提升。

摘要:

大语言模型(Large Language Models, LLMs)和预训练语言模型(pre-trained Language Models, LMs)已在众多软件工程任务(如代码补全、代码生成)中取得了令人瞩目的成果。借助GitHub等海量现有代码语料库,这类模型能够理解源代码中的模式,并利用这些模式预测代码属性。然而,LLM在少样本学习下于故障定位、漏洞定位等非生成类任务上表现不佳,且对终端用户和小型组织而言,微调LLM既耗时又成本高昂。此外,针对非生成类任务微调的LM性能虽表现出色,但其效果严重依赖数据的数量与质量。因此,当前现实场景中数据的匮乏以及数据收集的高成本进一步限制了LM的适用性。本文中,我们利用LLM强大的生成能力来增强预训练LM。具体而言,我们使用LLM生成领域特定数据,以此提升预训练LM在目标任务上的性能。我们在生成阶段结合不同的LLM,并引入各类LM从LLM生成的数据中学习,开展了相关实验,随后对比了这些LM学习该数据前后的性能。研究发现,LLM生成的数据能显著提升LM的性能:在故障定位任务上提升幅度最高可达58.36%,在克隆检测任务上提升幅度最高可达6.09%。

英文摘要:

Large Language Models (LLMs) and pre-trained Language Models (LMs) have achieved impressive success on many software engineering tasks (e.g., code completion and code generation). By leveraging huge existing code corpora (e.g., GitHub), these models can understand the patterns in source code and use these patterns to predict code properties. However, LLMs under few-shot learning perform poorly on non-generative tasks (e.g., fault localization and vulnerability localization), and fine-tuning LLMs is time-consuming and costly for end users and small organizations. Furthermore, the performance of fine-tuning LMs for non-generative tasks is impressive, yet it heavily depends on the amount and quality of data. As a result, the current lack of data and the high cost of collecting it in real-world scenarios further limit the applicability of LMs. In this paper, we leverage the powerful generation capabilities of LLMs to enhance pre-trained LMs. Specifically, we use LLMs to generate domain-specific data, thereby improving the performance of pre-trained LMs on the target tasks. We conduct experiments by combining different LLMs in our generation phase and introducing various LMs to learn from the LLM-generated data. Then, we compare the performance of these LMs before and after learning the data. We find that LLM-generated data significantly enhances the performance of LMs. The improvement can reach up to 58.36% for fault localization and up to 6.09% for clone detection.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑