当生活给你大语言模型时,制作LLM-ADE:具有自适应数据工程的大语言模型
When Life gives you LLMs, make LLM-ADE: Large Language Models with Adaptive Data Engineering
AI总结:
本文提出LLM-ADE框架,通过动态架构调整(选择性块冻结与扩展)实现大语言模型持续预训练,解决灾难性遗忘与双下降问题,在TinyLlama上显著提升性能。
摘要:
本文介绍了LLM-ADE框架,这是一种用于大语言模型(LLMs)持续预训练的新方法,解决了灾难性遗忘和双下降的挑战。LLM-ADE采用动态架构调整,包括选择性块冻结和扩展,针对特定数据集进行定制。该策略增强了模型对新数据的适应性,同时保留了先前获得的知识。我们在TinyLlama模型上展示了LLM-ADE在各种通用知识基准上的有效性,显示出显著的性能提升,而没有传统持续训练方法的缺点。这种方法为在现实应用中保持LLMs的时效性和效率提供了一种更加通用和稳健的方式。
英文摘要:
This paper presents the LLM-ADE framework, a novel methodology for continued pre-training of large language models (LLMs) that addresses the challenges of catastrophic forgetting and double descent. LLM-ADE employs dynamic architectural adjustments, including selective block freezing and expansion, tailored to specific datasets. This strategy enhances model adaptability to new data while preserving previously acquired knowledge. We demonstrate LLM-ADE's effectiveness on the TinyLlama model across various general knowledge benchmarks, showing significant performance improvements without the drawbacks of traditional continuous training methods. This approach promises a more versatile and robust way to keep LLMs current and efficient in real-world applications.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。