arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.13137cs.CL
收藏

Transformer 语言适配器的隐藏空间

The Hidden Space of Transformer Language Adapters

Mila, McGill University(米拉-麦吉尔大学);Google Research(谷歌研究院);Tel Aviv University(特拉维夫大学);Saarland University(萨尔大学)

Jesujoba O. Alabi, Marius Mosbach, Matan Eyal, Dietrich Klakow, Mor Geva

AI总结:

本文通过分析 transformer 语言适配器在冻结模型上的运行机制,发现适配过程主要依赖源语言表示、仅在最后几层显现目标语言且可跳过部分适配器,从而揭示了底层模型对适配的约束并提出了效率优化方向。

摘要:

我们分析了 transformer 语言适配器的运行机制,这些适配器是在冻结的语言模型之上训练的小型模块,用于将其预测适配到新的目标语言。我们表明,适配后的预测主要在模型训练所用的源语言中演化,而目标语言仅在模型的最后几层才变得明显。此外,适配过程是渐进的并分布在各个层中,其中可以跳过小组适配器而不会降低适配性能。最后,我们表明适配器在模型冻结的表示空间之上运行,同时在很大程度上保留其结构,而不是在“孤立”的子空间上运行。我们的发现为语言模型向新语言的适配过程提供了更深入的视角,展示了底层模型对其施加的约束,并引入了提高其效率的实践意义。

英文摘要:

We analyze the operation of transformer language adapters, which are small modules trained on top of a frozen language model to adapt its predictions to new target languages. We show that adapted predictions mostly evolve in the source language the model was trained on, while the target language becomes pronounced only in the very last layers of the model. Moreover, the adaptation process is gradual and distributed across layers, where it is possible to skip small groups of adapters without decreasing adaptation performance. Last, we show that adapters operate on top of the model's frozen representation space while largely preserving its structure, rather than on an 'isolated' subspace. Our findings provide a deeper view into the adaptation process of language models to new languages, showcasing the constraints imposed on it by the underlying model and introduces practical implications to enhance its efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑