arXiv 2301.054872024-04-04cs.CL

FUN with Fisher：利用计划解冻提升基于适配器的跨语言迁移泛化能力

FUN with Fisher: Improving Generalization of Adapter-Based Cross-lingual Transfer with Scheduled Unfreezing

Technical University of Darmstadt（达姆施塔特工业大学）；Hessian Center for AI (hessian.AI)（黑森人工智能中心）；Google DeepMind（谷歌DeepMind）；University of Cambridge（剑桥大学）

Chen Cecilia Liu, Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文研究基于适配器的跨语言迁移中的计划解冻算法，发现其能缩小与全量微调的差距并提升跨语言泛化性能，同时利用 Fisher Information 揭示其学习动态与泛化性能相关，并提出通用算法在四个数据集上平均提升 2 个点。

摘要：

语言模型的标准微调通常在分布内数据上表现良好，但在面对分布偏移时泛化能力会下降。在这项工作中，我们旨在提升基于适配器的跨语言任务迁移的泛化能力，因为在这种场景下跨语言分布偏移是不可避免的。我们研究了计划解冻算法——最初是为了缓解迁移学习中的灾难性遗忘而提出的——用于微调任务适配器。我们的实验表明，计划解冻方法缩小了与全量微调之间的差距，并实现了更强的跨语言迁移性能，说明这些方法的作用不仅限于缓解灾难性遗忘。接下来，为了理解这些实证发现，我们利用 Fisher Information 研究了计划解冻的学习动态。实验揭示，与标准微调相比，计划解冻会诱导出不同的学习动态，并提供证据表明训练过程中 Fisher Information 的动态与跨语言泛化性能相关。此外，我们提出了一个通用的计划解冻算法，在四个数据集上相比标准微调平均提升了 2 个点，并为适配器训练中启发式解冻计划的基于理论的合理性提供了实证证据。

英文摘要：

Standard fine-tuning of language models typically performs well on in-distribution data, but suffers with generalization to distribution shifts. In this work, we aim to improve the generalization of adapter-based cross-lingual task transfer where such cross-language distribution shifts are imminent. We investigate scheduled unfreezing algorithms -- originally proposed to mitigate catastrophic forgetting in transfer learning -- for fine-tuning task adapters. Our experiments show that scheduled unfreezing methods close the gap to full fine-tuning and achieve stronger cross-lingual transfer performance, suggesting that these methods can go beyond just mitigating catastrophic forgetting. Next, aiming to understand these empirical findings, we investigate the learning dynamics of scheduled unfreezing using Fisher Information. Our experiments reveal that scheduled unfreezing induces different learning dynamics compared to standard fine-tuning, and provide evidence that the dynamics of Fisher Information during training correlate with cross-lingual generalization performance. We additionally propose a general scheduled unfreezing algorithm that achieves an average of 2 points improvement over four datasets compared to standard fine-tuning and provides empirical evidence for a theory-based justification of the heuristic unfreezing schedule for adapter training.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。