增强低资源语言的代码生成:没有银弹
Enhancing Code Generation for Low-Resource Languages: No Silver Bullet
AI总结:
本研究实证评估了微调、上下文学习和跨语言翻译预训练三种方法在R和Racket等低资源语言代码生成中的效果,发现小模型微调最佳,大模型上下文学习更优,超大模型微调反而可能因数据不足导致性能下降。
摘要:
大型语言模型(LLMs)的出现显著推进了自动代码生成领域。LLMs 依赖大型且多样化的数据集来学习编程语言的语法、语义和使用模式。对于低资源语言(即以训练数据稀缺为特征的小众编程语言),此类数据的有限可用性阻碍了模型有效泛化的能力,导致其代码生成性能与高资源语言相比表现较差。出于这个原因,人们一直在寻求能够弥合这一性能差距的技术。我们提出了一项实证研究,调查了几种提升 LLMs 在低资源语言上性能的方法的有效性,即:(i) 经典的微调,然而由于训练数据的稀缺,其规模受到限制;(ii) 上下文学习的三个变体,通过精心设计的提示为 LLM 提供关于低资源语言的额外信息(例如,展示目标语言特性的少样本示例);以及 (iii) 一种预训练目标,教会模型如何在资源丰富和资源匮乏的语言之间进行翻译。我们研究的背景是两种低资源语言(R 和 Racket)以及六种具有不同架构和规模的 LLMs。我们的发现表明,对于较小的 LLMs,微调通常是最佳选择,这可能是因为即使是小型数据集也足以训练其有限数量的参数。随着模型规模的增加,上下文学习变得越来越有效,代表了一种安全且低成本的选择(即它总是有帮助的,但程度不同)。不同的是,对于超大型 LLMs,在进行微调时可能会降低其在低资源语言上的性能,这可能是由于缺乏足够的数据来有效更新其权重。
英文摘要:
The advent of Large Language Models (LLMs) has significantly advanced the field of automated code generation. LLMs rely on large and diverse datasets to learn syntax, semantics, and usage patterns of programming languages. For low-resource languages (i.e., niche programming languages characterized by the scarcity of training data), the limited availability of such data hampers the models' ability to generalize effectively, resulting in poorer code generation performance as compared to high-resource languages. For this reason, there is a quest for techniques able to close this performance gap. We present an empirical study investigating the effectiveness of several approaches for boosting LLMs' performance on low-resource languages, namely: (i) a classic fine-tuning, which is however capped in size by the scarcity of training data; (ii) three variants of in-context learning, with prompts crafted to provide the LLM with additional information about the low-resource language (e.g., few-shot examples showcasing features of the targeted language); and (iii) a pre-training objective teaching the model how to translate between high- and low-resource languages. The context of our study are two low-resource languages (R and Racket) and six LLMs having different architectures and sizes. Our findings reveal that a fine-tuning is usually the best choice for smaller LLMs, possibly due to the fact that even a small dataset is sufficient to train their limited number of parameters. With the increase in size of the models, in-context learning becomes more and more effective, representing a safe and cheap bet (i.e., it always helps, but with different magnitudes). Differently, very large LLMs may deteriorate their performance on low-resource languages when fine-tuning is performed, possibly due to the lack of enough data needed to effectively update their weights.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。