arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.02193cs.CLcs.LG
收藏

CodeACT:面向代码大语言模型的代码自适应计算高效微调框架

CodeACT: Code Adaptive Compute-efficient Tuning Framework for Code LLMs

Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Shenzhen Institute of Artificial Intelligence and Robotics for Society(深圳市人工智能与机器人研究院)

Weijie Lv, Xuan Xia, Sheng-Jun Huang

AI总结:

本文提出CodeACT框架,通过CDAS高质量数据选择与Dynamic Pack填充策略,使6.7B代码模型仅用40%数据即可提升性能并大幅降低训练时间与显存。

摘要:

大语言模型(LLM)在代码相关任务中展现出巨大潜力,但开源模型仍落后于闭源模型。为弥合这一性能差距,现有方法会生成大量合成数据进行微调,从而导致训练效率低下。受更有效、更高效训练需求的启发,我们提出了代码自适应计算高效微调(Code Adaptive Compute-efficient Tuning,CodeACT)框架。CodeACT引入了复杂度与多样性感知采样(Complexity and Diversity Aware Sampling,CDAS)方法,以根据复杂度和多样性选择高质量训练数据;并采用Dynamic Pack填充策略,通过在训练期间最小化填充标记来减少计算资源使用。实验结果表明,仅在40%的EVOL-Instruct数据上进行微调的CodeACT-DeepSeek-Coder-6.7B,在HumanEval上实现了8.6%的性能提升,训练时间减少78%,峰值GPU内存占用降低27%。这些发现凸显了CodeACT增强开源模型性能和效率的能力。通过同时优化数据选择和训练流程,CodeACT提供了一种综合方法,可在显著降低计算需求的同时提升开源LLM的能力,应对数据质量和训练效率的双重挑战,并为更节省资源、性能更强的模型铺平道路。

英文摘要:

Large language models (LLMs) have shown great potential in code-related tasks, yet open-source models lag behind their closed-source counterparts. To bridge this performance gap, existing methods generate vast amounts of synthetic data for fine-tuning, leading to inefficiencies in training. Motivated by the need for more effective and efficient training, we propose the Code Adaptive Compute-efficient Tuning (CodeACT) framework. CodeACT introduces the Complexity and Diversity Aware Sampling (CDAS) method to select high-quality training data based on complexity and diversity, and the Dynamic Pack padding strategy to reduce computational resource usage by minimizing padding tokens during training. Experimental results demonstrate that CodeACT-DeepSeek-Coder-6.7B, fine-tuned on only 40% of the EVOL-Instruct data, achieves an 8.6% performance increase on HumanEval, reduces training time by 78%, and decreases peak GPU memory usage by 27%. These findings underscore CodeACT's ability to enhance the performance and efficiency of open-source models. By optimizing both the data selection and training processes, CodeACT offers a comprehensive approach to improving the capabilities of open-source LLMs while significantly reducing computational requirements, addressing the dual challenges of data quality and training efficiency, and paving the way for more resource-efficient and performant models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑