发表机构
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究探究LoRA微调LLMs用于重排序时，注意力更新在网络中间区域的作用，发现其与公理IR特征关注度相关，为改进重排序器适配策略提供了可解释依据。
AI 中文摘要
LoRA微调是将大语言模型（LLMs）适配到重排序任务的标准方法，但目前仍不清楚网络中特定任务的相关性行为在何处被学习，以及该学习过程伴随哪些注意力层面的变化。通过消融实验和注意力实验，我们确定了LoRA对RankLLaMA的注意力更新在何处提升性能，以及这些性能提升是否与可解释的面向相关性的注意力模式（如词汇匹配、稀有性敏感性、查询-文档交互）重合。我们发现，在整个网络中存在经LoRA微调的多层感知机（MLPs）的前提下，将LoRA注意力更新限制在紧凑的网络中间区域，足以恢复将LoRA应用于所有注意力层所获得的超过一半的性能，且忽略该区域的注意力微调对性能的损害大于网络其他区域。此外，我们表明，应用LoRA对性能影响最大的区域，与微调时注意力对公理信息检索（IR）特征关注度增加的区域重合。稀有性敏感性、文档-查询交互以及若干组合特征，与排序性能的提升高度相关。我们的结果支持一种可解释的、相关性的观点，即面向相关性的行为在LoRA微调期间如何涌现，并为改进重排序器的适配策略指明了方向。
英文摘要
LoRA fine-tuning is standard for adapting LLMs to reranking, but it remains unclear where in the network task-specific relevance behavior is learned and what attention-level changes accompany that learning. Through ablation and attention experiments, we identify where LoRA attention updates to RankLLaMA improve performance and whether those gains coincide with interpretable relevance-oriented attention patterns such as lexical matching, rarity sensitivity, and query-document interaction. We find that given LoRA fine-tuned MLPs throughout the network, restricting LoRA attention updates to a compact mid-network region is sufficient for recovering over half of the performance gained by applying LoRA to all attention layers, and that omitting attention fine-tuning in this region hurts performance more than elsewhere in the network. Additionally, we show that regions where applying LoRA affects performance the most overlap with regions where fine-tuning increased attention to axiomatic IR features. Rarity sensitivity, document-query interaction, and several compositional features are highly correlated with gains in ranking performance. Our results support an interpretable, correlational account of how relevance-oriented behavior emerges during LoRA fine-tuning and point toward improved strategies for adapting rerankers.
CommentsAccepted to EMNLP 2026 Findings. 17 Pages. 25 Figures. 5 Tables
