面向低资源任务的领域自适应持续学习：在尼泊尔语上的评估
Domain-adaptative Continual Learning for Low-resource Tasks: Evaluation on Nepali
AI总结：
本研究评估了在低资源尼泊尔语环境下，使用合成数据通过4-bit QLoRA持续训练Llama 3 8B的领域自适应预训练（DAPT）可行性，发现模型虽有遗忘但在多shot评估中表现出潜在的记忆保留能力。
摘要：
由于在新数据可用时从头重新训练大型语言模型（LLMs）不可行，持续学习已成为一个重要的研究方向。领域自适应预训练（DAPT）范式备受关注，其重点在于持续训练预训练语言模型，使其适应最初未训练过的领域。在本研究中，我们评估了 DAPT 在低资源环境（即尼泊尔语）中的可行性。我们使用合成数据继续训练 Llama 3 8B，在 4-bit QLoRA 设置下使其适应尼泊尔语。我们从性能、遗忘和知识获取三个方面评估了适配后的模型。我们比较了基础模型和最终模型在尼泊尔语生成能力及在流行基准上的表现，并进行了案例研究以探究它们在尼泊尔语上的语言知识。我们在最终模型中观察到了意料之中的遗忘，但也令人惊讶地发现，在评估期间增加 shot 数量，最终模型获得了更高的百分比提升（高达 19.29%），而基础模型为 4.98%，这表明存在潜在的记忆保留。我们还探索了层-头自注意力热图，以确立最终模型在尼泊尔语中的依赖解析能力。
英文摘要：
Continual learning has emerged as an important research direction due to the infeasibility of retraining large language models (LLMs) from scratch in the event of new data availability. Of great interest is the domain-adaptive pre-training (DAPT) paradigm, which focuses on continually training a pre-trained language model to adapt it to a domain it was not originally trained on. In this work, we evaluate the feasibility of DAPT in a low-resource setting, namely the Nepali language. We use synthetic data to continue training Llama 3 8B to adapt it to the Nepali language in a 4-bit QLoRA setting. We evaluate the adapted model on its performance, forgetting, and knowledge acquisition. We compare the base model and the final model on their Nepali generation abilities, their performance on popular benchmarks, and run case-studies to probe their linguistic knowledge in Nepali. We see some unsurprising forgetting in the final model, but also surprisingly find that increasing the number of shots during evaluation yields better percent increases in the final model (as high as 19.29% increase) compared to the base model (4.98%), suggesting latent retention. We also explore layer-head self-attention heatmaps to establish dependency resolution abilities of the final model in Nepali.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
