arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节(Veterans Day),12 日无更新
arXiv 2602.12192cs.CL
收藏

面向长上下文处理的查询聚焦与记忆感知重排序器

Query-focused and Memory-aware Reranker for Long Context Processing

Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Pattern Recognition Center, WeChat AI, Tencent(腾讯微信人工智能研究院模式识别中心);East China Normal University(华东师范大学)

Yuqing Li, Jiangnan Li, Mo Yu, Guoxuan Ding, Yanyu Chen, Zheng Lin, Wei Zhang, Jie Zhou

AI总结:

提出一种基于大语言模型中检索头注意力分数的列表式重排序框架,利用候选列表整体信息估计段落-查询相关性,无需Likert监督,在多个领域取得最优性能。

摘要:

基于现有对大语言模型中检索头的分析,我们提出了一种替代的重排序框架,该框架训练模型使用选定头的注意力分数来估计段落-查询相关性。这种方法提供了一种列表式解决方案,在排序过程中利用整个候选短列表中的整体信息。同时,它自然地产生连续的相关性分数,使得能够在任意检索数据集上进行训练,而无需Likert量表监督。我们的框架轻量且有效,仅需小规模模型(如3B参数)即可实现强性能。大量实验表明,我们的方法在多个领域(包括维基百科和长叙事数据集)中优于现有的最先进点式和列表式重排序器。它进一步在评估对话理解和记忆使用的LoCoMo基准上建立了新的最先进水平。我们还证明我们的框架支持灵活的扩展。例如,用上下文信息增强候选段落可进一步提高排序准确性,而训练中间层的注意力头可在不牺牲性能的情况下提高效率。

英文摘要:

Built upon the existing analysis of retrieval heads in large language models, we propose an alternative reranking framework that trains models to estimate passage-query relevance using the attention scores of selected heads. This approach provides a listwise solution that leverages the holistic information within the entire candidate shortlist during ranking. At the same time, it naturally produces continuous relevance scores, enabling training on arbitrary retrieval datasets without requiring Likert-scale supervision. Our framework is lightweight and effective, requiring only small-scale models, such as 3B parameters, to achieve strong performance. Extensive experiments demonstrate that our method outperforms existing state-of-the-art pointwise and listwise rerankers across multiple domains, including Wikipedia and long narrative datasets. It further establishes a new state-of-the-art on the LoCoMo benchmark, which assesses dialogue understanding and memory usage. We further demonstrate that our framework supports flexible extensions. For example, augmenting candidate passages with contextual information further improves ranking accuracy, while training attention heads from middle layers enhances efficiency without sacrificing performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑