定向词汇注入：通过早期层LoRA微调解锁Lugha-Llama中的潜在跨语言对齐
Targeted Lexical Injection: Unlocking Latent Cross-Lingual Alignment in Lugha-Llama via Early-Layer LoRA Fine-Tuning
AI总结：
针对低资源语言模型词汇对齐不足的问题，提出定向词汇注入（TLI）方法，利用早期层潜在对齐能力，通过LoRA和对比学习微调，显著提升斯瓦希里语-英语词对的对齐性能并泛化到未见词对。
摘要：
大型语言模型（LLMs）已展现出卓越的能力，然而它们在低资源语言（LRLs）如斯瓦希里语上的表现往往因预训练中的数据稀缺和代表性不足而落后。一个关键挑战是实现稳健的跨语言词汇对齐，这对于翻译和跨语言信息检索等任务至关重要。本文介绍了定向词汇注入（TLI），一种新颖且高效的微调方法。我们首先证明Lugha-Llama-8B-wura（一个以斯瓦希里语为中心的LLM）在其早期内部层（具体为第2层，基于试点研究的平均余弦相似度约为0.99998）中对斯瓦希里语-英语词对表现出强大、近乎完美的词汇对齐，而这种能力并未完全反映在其最终输出表示中（在我们的评估集上基线相似度约为0.32）。TLI利用这一见解，通过使用低秩适配（LoRA）和对比学习目标对模型进行微调，专门针对这一经验确定的最佳早期层的嵌入。我们的实验表明，TLI显著提高了623个训练过的斯瓦希里语-英语词对的输出级词汇对齐，将平均余弦相似度从0.3211提升至0.4113（+28.08%，p < 1.33 x 10^-240）。更重要的是，这些改进对63个未见过的对照词对也表现出显著的泛化能力，相似度从0.3143提升至0.4033（+28.32%，p < 7.17 x 10^-27）。这些发现表明，TLI增强了模型保留和传播其固有早期层跨语言知识的能力，为改善以低资源语言为重点的LLMs中的词汇对齐提供了一种参数高效且有效的策略。
英文摘要：
Large Language Models (LLMs) have demonstrated remarkable capabilities, yet their performance in low-resource languages (LRLs), such as Swahili, often lags due to data scarcity and underrepresentation in pre-training. A key challenge is achieving robust cross-lingual lexical alignment, crucial for tasks like translation and cross-lingual information retrieval. This paper introduces Targeted Lexical Injection (TLI), a novel and efficient fine-tuning approach. We first demonstrate that Lugha-Llama-8B-wura, a Swahili-centric LLM, exhibits strong, near-perfect lexical alignment for Swahili-English word pairs in its early internal layers (specifically Layer 2, with ~0.99998 average cosine similarity based on a pilot study), a capability not fully reflected in its final output representations (baseline ~0.32 similarity on our evaluation set). TLI leverages this insight by using Low-Rank Adaptation (LoRA) and a contrastive learning objective to fine-tune the model, specifically targeting embeddings from this empirically identified optimal early layer. Our experiments show that TLI significantly improves the output-level lexical alignment for 623 trained Swahili-English word pairs, increasing average cosine similarity from 0.3211 to 0.4113 (+28.08%, p < 1.33 x 10^-240). More importantly, these improvements generalize remarkably well to 63 unseen control word pairs, with similarity increasing from 0.3143 to 0.4033 (+28.32%, p < 7.17 x 10^-27). These findings suggest TLI enhances the model's ability to preserve and propagate its inherent early-layer cross-lingual knowledge, offering a parameter-efficient and effective strategy for improving lexical alignment in LRL-focused LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
