arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.04880cs.AI
收藏

DMA:基于人类反馈的在线 RAG 对齐

DMA: Online RAG Alignment with Human Feedback

Zhongguancun Laboratory(中关村实验室);Tsinghua University(清华大学);China Mobile Communications Group Co., Ltd.(中国移动通信集团有限公司);Beijing University of Posts and Telecommunications(北京邮电大学)

Yu Bai, Yukai Miao, Dawei Wang, Li Chen, Fei Long, Rundi Zhai, Dan Li, Yanyu Ren, Tianfeng Liu, Hongtao Xie, Ce Yang, Xuhui Cai

AI总结:

本文提出动态记忆对齐(DMA)框架,通过融合文档级、列表级和响应级人类反馈,结合监督训练、策略优化与知识蒸馏,实现 RAG 排序的在线实时对齐,并在在线部署和离线问答基准上验证了其在不损失基线能力前提下的显著增益。

摘要:

检索增强生成(RAG)系统通常依赖静态检索,这限制了其对不断演变的意图和内容漂移的适应能力。我们提出动态记忆对齐(Dynamic Memory Alignment,DMA),这是一种在线学习框架,系统性地融合多粒度人类反馈,以在交互式场景中对排序进行对齐。DMA 将文档级、列表级和响应级信号组织成一个连贯的学习流程:针对逐点(pointwise)和逐列表(listwise)排序器的监督训练、由响应级偏好驱动的策略优化,以及将知识蒸馏到轻量级打分器中以实现低延迟服务。在本文中,记忆指的是模型的工作记忆,即大语言模型(LLM)在进行上下文学习(In-Context Learning)时可见的全部上下文。我们采用双轨评估协议以模拟真实部署:(i)大规模在线 A/B 消融实验,用于隔离每个反馈来源的效用;以及(ii)在知识密集型基准上的少样本离线测试。在线方面,一项持续数月的工业部署进一步显示出人类参与度的显著提升。离线方面,DMA 在保持具有竞争力的基础检索能力的同时,在对话式问答(TriviaQA、HotpotQA)上取得了显著增益。综合来看,这些结果表明 DMA 是一种有原则的方法,能够在 RAG 中实现由反馈驱动的实时适应,同时不牺牲基线能力。

英文摘要:

Retrieval-augmented generation (RAG) systems often rely on static retrieval, limiting adaptation to evolving intent and content drift. We introduce Dynamic Memory Alignment (DMA), an online learning framework that systematically incorporates multi-granularity human feedback to align ranking in interactive settings. DMA organizes document-, list-, and response-level signals into a coherent learning pipeline: supervised training for pointwise and listwise rankers, policy optimization driven by response-level preferences, and knowledge distillation into a lightweight scorer for low-latency serving. Throughout this paper, memory refers to the model's working memory, which is the entire context visible to the LLM for In-Context Learning. We adopt a dual-track evaluation protocol mirroring deployment: (i) large-scale online A/B ablations to isolate the utility of each feedback source, and (ii) few-shot offline tests on knowledge-intensive benchmarks. Online, a multi-month industrial deployment further shows substantial improvements in human engagement. Offline, DMA preserves competitive foundational retrieval while yielding notable gains on conversational QA (TriviaQA, HotpotQA). Taken together, these results position DMA as a principled approach to feedback-driven, real-time adaptation in RAG without sacrificing baseline capability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑