DeFTX: 面向零样本跨语言迁移的去噪稀疏微调
DeFTX: Denoised Sparse Fine-Tuning for Zero-Shot Cross-Lingual Transfer
AI总结：
针对跨语言迁移挑战，该研究提出DeFT-X方法，通过奇异值分解对预训练模型权重去噪后再做幅值剪枝，得到更鲁棒的可组合稀疏微调向量，在极低资源语言任务上性能优于或持平现有基线。
摘要：
实现有效的跨语言迁移，仍是将大语言模型的优势从高资源语言拓展到低资源语言过程中的关键挑战。为达成这一目标，已有研究探索了多种方法，以结合来自（高资源）源语言任务特定数据的任务知识，与来自（低资源）目标语言无标注文本的语言知识。其中一种值得关注的方法是用于跨语言迁移的可组合稀疏微调（SFT），该方法学习任务特定与语言特定的稀疏掩码，以选出预训练模型的一部分参数进行进一步微调。这些稀疏微调向量（SFT）随后与预训练模型组合，仅使用源语言的任务特定数据，即可实现向目标语言任务的零样本跨语言迁移。此前SFT的稀疏掩码是通过简单的基于幅值的剪枝确定的。在本研究中，我们提出了DeFT-X这一新型可组合SFT方法，它在基于幅值的剪枝前，利用奇异值分解对预训练模型的权重矩阵进行去噪，从而得到更鲁棒的SFT。我们在多种极低资源语言的情感分类任务（NusaX）和自然语言推理任务（AmericasNLI）上对DeFT-X进行了评估，结果表明其性能与SFT及其他主流跨语言迁移基线相当，甚至更优。
英文摘要：
Effective cross-lingual transfer remains a critical challenge in scaling the benefits of large language models from high-resource to low-resource languages. Towards this goal, prior studies have explored many approaches to combine task knowledge from task-specific data in a (high-resource) source language and language knowledge from unlabeled text in a (low-resource) target language. One notable approach proposed composable sparse fine-tuning (SFT) for cross-lingual transfer that learns task-specific and language-specific sparse masks to select a subset of the pretrained model's parameters that are further fine-tuned. These sparse fine-tuned vectors (SFTs) are subsequently composed with the pretrained model to facilitate zero-shot cross-lingual transfer to a task in a target language, using only task-specific data from a source language. These sparse masks for SFTs were identified using a simple magnitude-based pruning. In our work, we introduce DeFT-X, a novel composable SFT approach that denoises the weight matrices of a pretrained model before magnitude pruning using singular value decomposition, thus yielding more robust SFTs. We evaluate DeFT-X on a diverse set of extremely low-resource languages for sentiment classification (NusaX) and natural language inference (AmericasNLI) and demonstrate that it performs at par or outperforms SFT and other prominent cross-lingual transfer baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
