arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.05064cs.CL
收藏

跨语言模型的阶数级注意力相似性:一种潜在共性

Order-Level Attention Similarity Across Language Models: A Latent Commonality

South China University of Technology(华南理工大学);Pazhou Laboratory(Pazhou实验室);Zhuzhou CRRC Times Electric Co.(株洲 CRRC 时代电子公司);China Telecom Research Institute(中国电信研究院);WeBank(我们银行);The Hong Kong University of Science and Technology(香港科技大学)

Jinglin Liang, Jin Zhong, Shuangping Huang, Yunqing Hu, Huiyuan Zhang, Huifang Li, Lixin Fan, Hanlin Gu

AI总结:

本文提出阶数级注意力(OLA)并发现其在不同语言模型间具有显著相似性且与句法知识存在隐式映射,据此设计无需训练的跨模型适配器迁移方法TOA,有效提升未见语言模型的性能。

摘要:

本文探讨了一个重要但此前被忽视的问题:语言模型(LMs)之间的上下文聚合模式是否具有共性?尽管已有一些工作研究了语言模型中的上下文聚合或注意力权重,但它们通常聚焦于单个模型或单个注意力头,缺乏跨多个语言模型的系统性分析来探索其共性。与此不同,我们关注语言模型之间的共性,这有助于加深对语言模型的理解,甚至促进跨模型知识迁移。在本工作中,我们引入基于Attention Rollout逐阶分解得到的阶数级注意力(OLA),并揭示不同语言模型在相同阶数上的OLA表现出显著相似性。此外,我们发现OLA与句法知识之间存在一种隐式映射关系。基于这两项发现,我们提出了可迁移OLA适配器(TOA),一种无需训练的跨语言模型适配器迁移方法。具体而言,我们将OLA视为统一的句法特征表示,并训练一个以OLA为输入的适配器。由于不同语言模型之间OLA的相似性,该适配器能够泛化到未见过的语言模型,而无需任何参数更新。大量实验表明,TOA的跨语言模型泛化能力有效提升了未见语言模型的性能。代码可在 https://github.com/jinglin-liang/OLAS 获取。

英文摘要:

In this paper, we explore an important yet previously neglected question: Do context aggregation patterns across Language Models (LMs) share commonalities? While some works have investigated context aggregation or attention weights in LMs, they typically focus on individual models or attention heads, lacking a systematic analysis across multiple LMs to explore their commonalities. In contrast, we focus on the commonalities among LMs, which can deepen our understanding of LMs and even facilitate cross-model knowledge transfer. In this work, we introduce the Order-Level Attention (OLA) derived from the order-wise decomposition of Attention Rollout and reveal that the OLA at the same order across LMs exhibits significant similarities. Furthermore, we discover an implicit mapping between OLA and syntactic knowledge. Based on these two findings, we propose the Transferable OLA Adapter (TOA), a training-free cross-LM adapter transfer method. Specifically, we treat the OLA as a unified syntactic feature representation and train an adapter that takes OLA as input. Due to the similarities in OLA across LMs, the adapter generalizes to unseen LMs without requiring any parameter updates. Extensive experiments demonstrate that TOA's cross-LM generalization effectively enhances the performance of unseen LMs. Code is available at https://github.com/jinglin-liang/OLAS.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑