基于前缀适配的零样本跨语言迁移
Zero-Shot Cross-Lingual Transfer using Prefix-Based Adaptation
AI总结：
本研究针对仅解码器大语言模型跨语言任务适配难题，综合探究三种前缀式参数高效方法的零样本跨语言迁移效果，证实其在多基准上优于LoRA基线，为低资源多语言场景提供了高效可扩展的适配方案。
摘要：
随着Llama、Mistral等新型大语言模型（LLM）的发布，得益于其多语言预训练和强大的泛化能力，零样本跨语言迁移变得愈发可行。然而，让这些仅解码器架构的LLM适配跨语言的新任务仍具挑战。尽管低秩适配（LoRA）等参数高效微调（PeFT）技术已被广泛应用，但软提示调优、前缀调优、Llama Adapter等基于前缀的技术探索仍较少，尤其是在仅解码器模型的零样本迁移场景中。我们针对三种基于前缀的方法开展了综合研究，探究其从英语到35种以上高、低资源语言的零样本跨语言迁移效果。我们的分析进一步探索了跨语系和书写系统的迁移效果，以及模型规模从1B到24B缩放带来的影响。在Belebele基准测试上，使用Llama 3.1 8B时，前缀方法的表现比LoRA基线最高高出6%；使用Mistral v0.3 7B时也观测到了类似的提升。尽管前缀调优仅使用123万个可学习参数，我们仍在多个不同基准上取得了稳定提升。这些发现凸显了基于前缀的技术作为LoRA有效且可扩展替代方案的潜力，尤其适用于低资源多语言场景。
英文摘要：
With the release of new large language models (LLMs) like Llama and Mistral, zero-shot cross-lingual transfer has become increasingly feasible due to their multilingual pretraining and strong generalization capabilities. However, adapting these decoder-only LLMs to new tasks across languages remains challenging. While parameter-efficient fine-tuning (PeFT) techniques like Low-Rank Adaptation (LoRA) are widely used, prefix-based techniques such as soft prompt tuning, prefix tuning, and Llama Adapter are less explored, especially for zero-shot transfer in decoder-only models. We present a comprehensive study of three prefix-based methods for zero-shot cross-lingual transfer from English to 35+ high- and low-resource languages. Our analysis further explores transfer across linguistic families and scripts, as well as the impact of scaling model sizes from 1B to 24B. With Llama 3.1 8B, prefix methods outperform LoRA-baselines by up to 6% on the Belebele benchmark. Similar improvements were observed with Mistral v0.3 7B as well. Despite using only 1.23M learning parameters with prefix tuning, we achieve consistent improvements across diverse benchmarks. These findings highlight the potential of prefix-based techniques as an effective and scalable alternative to LoRA, particularly in low-resource multilingual settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
