arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.07023cs.CLcs.AI
收藏

通过偏好对齐学习长上下文的推理

Hit-RAG: Learning to Reason with Long Contexts via Preference Alignment

Tongji University(同济大学);The City University of New York(纽约城市大学);University of Technology Sydney(悉尼大学);Huazhong University of Science and Technology(华中科技大学);Shenzhen University of Advanced Technology(深圳先进技术大学)

Junming Liu, Yuqi Li, Shiping Wen, Zhigang Zeng, Tingwen Huang

AI总结:

Hit-RAG通过多阶段偏好对齐框架解决长上下文推理中的注意力稀释和幻觉问题,提升模型在长上下文场景下的推理能力。

摘要:

尽管检索增强生成在将外部知识融入多模态大语言模型中展现出潜力,但向大量上下文过渡往往导致显著的注意力稀释和推理幻觉。信息密度的增加使关键证据被大量噪声淹没,这使得在密集输入中辨别相关片段变得更加复杂。在本文中,我们提出了Hit-RAG,一种多阶段偏好对齐框架,旨在通过渐进优化流程解决这些认知瓶颈。我们的方法系统地通过三个不同的阶段优化外部证据的利用。首先,监督微调建立基础的上下文意识以最小化信息遗漏。其次,判别偏好对齐增强对误导性干扰项的鲁棒性。最后,组相对策略优化稳定逻辑综合以防止推理崩溃。在八个基准测试中的广泛评估表明,Hit-RAG始终产生显著的性能提升,使模型能够弥合上下文获取与准确推理之间的差距,同时在长上下文场景中超越许多更大的模型。

英文摘要:

Despite the promise of Retrieval-Augmented Generation in grounding Multimodal Large Language Models with external knowledge, the transition to extensive contexts often leads to significant attention dilution and reasoning hallucinations. The surge in information density causes critical evidence to be submerged by voluminous noise, which complicates the discernment of relevant fragments within a dense input. In this paper, we propose \textbf{Hit-RAG}, a multi-stage preference alignment framework designed to resolve these cognitive bottlenecks through a progressive optimization pipeline. Our approach systematically refines the utilization of external evidence via three distinct stages. First, Supervised Fine-tuning establishes baseline context awareness to minimize information neglect. Next, Discriminative Preference Alignment enhances robustness against misleading distractors. Finally, Group-Relative Policy Optimization stabilizes logical synthesis to prevent reasoning collapse. Extensive evaluations on eight benchmarks demonstrate that Hit-RAG consistently yields substantial performance gains, enabling models to bridge the gap between context acquisition and accurate reasoning while surpassing much larger counterparts in long-context scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑