Gemma2 的全参数持续预训练：对流利度与领域知识的洞察
Full-Parameter Continual Pretraining of Gemma2: Insights into Fluency and Domain Knowledge
AI总结：
本文研究 Gemma2 在立陶宛语上的全参数持续预训练，通过 EWC 正则化防止灾难性遗忘，实验证明该方法能在保持领域知识的同时提升新语言流利度，为低资源语言高效适配提供参考。
摘要：
在本技术报告中，我们实证研究了在大语言模型（LLMs）持续学习背景下，语言流利度与领域知识之间的关系。具体而言，我们通过在 CulturaX 数据集的立陶宛语部分的前 10% 上自回归预训练 Gemma2 大语言模型的全部参数，来增强其立陶宛语的语言流利度。为了防止模型现有领域知识发生灾难性遗忘，我们应用了弹性权重巩固（EWC），利用从大规模多任务语言理解（MMLU）基准数据中估计的 Fisher 信息。在训练后评估中，我们通过困惑度评估语言流利度，并使用一系列语言理解基准（包括 ARC-Easy、Belebele、GSM8K、HellaSwag、MMLU、TruthfulQA 和 Winogrande）在英语和立陶宛语上的准确率来评估领域知识。实证结果表明，EWC 不仅通过保持模型在语言流利度和领域知识方面的表现来缓解灾难性遗忘，而且还改善或维持了新加入的立陶宛语的这些能力。这些发现凸显了通用大语言模型在无需访问原始训练数据的情况下更高效地适应代表性不足语言的潜力。随附的代码库可在 https://github.com/Neurotechnology/LLM_EWC 公开获取。
英文摘要：
In this technical report, we empirically investigate the relationship between linguistic fluency and domain knowledge in the context of continual learning with large language models (LLMs). Specifically, we enhance the linguistic fluency of the Gemma2 LLM for the Lithuanian language by autoregressively pretraining its full parameter set on the first 10\% of the Lithuanian language component of the CulturaX dataset. To prevent catastrophic forgetting of the model's existing domain knowledge, we apply Elastic Weight Consolidation (EWC), leveraging Fisher information estimated using data from the Massive Multitask Language Understanding (MMLU) benchmark. In the post-training evaluations, we assess linguistic fluency through perplexity and evaluate domain knowledge using accuracy on a suite of language understanding benchmarks, including ARC-Easy, Belebele, GSM8K, HellaSwag, MMLU, TruthfulQA, and Winogrande, in both English and Lithuanian. The empirical results demonstrate that EWC not only mitigates catastrophic forgetting by preserving the model's performance in terms of both linguistic fluency and domain knowledge but also improves or maintains these capabilities for the newly added Lithuanian language. These findings highlight the potential for more efficient adaptation of general-purpose LLMs to under-represented languages without requiring access to the original training data. The accompanying codebase is openly accessible at https://github.com/Neurotechnology/LLM_EWC.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
