简洁是智慧的灵魂:为代码生成剪枝长文件
Brevity is the soul of wit: Pruning long files for code generation
AI总结:
本文针对代码生成LLM的数据剪枝问题,发现基于嵌入的方法易受长度干扰,提出剪枝长文件的简单启发式方法,在计算受限条件下可提升训练效率或HumanEval性能,并引发对现有评估基准的反思。
摘要:
数据策展通常被视为大语言模型(LLM)训练的“秘密配方”,更高质量的数据通常能带来更好的LLM性能。鉴于互联网抓取语料库的规模,数据剪枝已成为越来越受关注的焦点。具体而言,许多研究表明,对数据进行去重或子选择更高质量的数据,可以带来效率或性能的提升。通常,过滤互联网规模语料库的方法有三类:基于嵌入的方法、基于启发式的方法和基于分类器的方法。在本工作中,我们在微调LLM用于代码生成的领域中对比了前两种方法。我们发现基于嵌入的方法常常受到长度因素的混淆,而一种简单的启发式方法——剪枝长文件——在计算受限的条件下优于其他方法。我们的方法在训练中可带来高达2倍的效率收益(同时保持性能匹配),或在HumanEval上带来3.5%的绝对性能提升(同时保持计算量匹配)。然而,我们发现保留的长文件上的困惑度可能会上升,这引出了一个问题:针对常见编码基准(HumanEval、MBPP)优化数据混合是否真正最有利于下游使用场景。总体而言,我们希望我们的工作能为代码数据(特别是极长代码文件的低质量)建立有用的直觉,提供一种有说服力的基于启发式的数据剪枝方法,并揭示我们在评估代码生成模型时面临的问题。
英文摘要:
Data curation is commonly considered a "secret-sauce" for LLM training, with higher quality data usually leading to better LLM performance. Given the scale of internet-scraped corpora, data pruning has become a larger and larger focus. Specifically, many have shown that de-duplicating data, or sub-selecting higher quality data, can lead to efficiency or performance improvements. Generally, three types of methods are used to filter internet-scale corpora: embedding-based, heuristic-based, and classifier-based. In this work, we contrast the former two in the domain of finetuning LLMs for code generation. We find that embedding-based methods are often confounded by length, and that a simple heuristic--pruning long files--outperforms other methods in compute-limited regimes. Our method can yield up to a 2x efficiency benefit in training (while matching performance) or a 3.5% absolute performance improvement on HumanEval (while matching compute). However, we find that perplexity on held-out long files can increase, begging the question of whether optimizing data mixtures for common coding benchmarks (HumanEval, MBPP) actually best serves downstream use cases. Overall, we hope our work builds useful intuitions about code data (specifically, the low quality of extremely long code files) provides a compelling heuristic-based method for data pruning, and brings to light questions in how we evaluate code generation models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。