少写代码，多对齐：通过数据剪枝实现高效的代码生成 LLM 微调
Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruning
AI总结：
本文针对代码 LLM 合成训练数据冗余问题，提出结合聚类与剪枝指标的数据筛选方法，仅用 10% 数据即可保持基准性能，适度剪枝还能提升生成质量并降低计算开销。
摘要：
近期针对代码生成大型语言模型（LLM）的研究表明，通过合成代码生成增加训练数据量通常会带来卓越性能。本文探索旨在提升代码 LLM 模型训练效率的数据剪枝方法。我们提出结合多种聚类与剪枝指标的技术，以选择性缩减训练数据，同时不损害生成代码的准确性和功能性。我们观察到合成训练数据生成中存在显著冗余；实验表明，仅使用 10% 的数据进行训练，即可在很大程度上保留基准测试性能。此外，我们发现对训练数据进行适度剪枝能够持续改善基准测试结果。实验表明，这些剪枝策略不仅减少了所需计算资源，还提升了代码生成的整体质量。
英文摘要：
Recent work targeting large language models (LLMs) for code generation demonstrated that increasing the amount of training data through synthetic code generation often leads to exceptional performance. In this paper we explore data pruning methods aimed at enhancing the efficiency of model training specifically for code LLMs. We present techniques that integrate various clustering and pruning metrics to selectively reduce training data without compromising the accuracy and functionality of the generated code. We observe significant redundancies in synthetic training data generation, where our experiments demonstrate that benchmark performance can be largely preserved by training on only 10% of the data. Moreover, we observe consistent improvements in benchmark results through moderate pruning of the training data. Our experiments show that these pruning strategies not only reduce the computational resources needed but also enhance the overall quality code generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
