arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.11342cs.LGcs.CL
收藏

解锁检索增强生成在扩散语言模型中的潜力

Unlocking the Potentials of Retrieval-Augmented Generation for Diffusion Language Models

Nankai University(南开大学);Beihang University(北航);HKUST (Guangzhou)(香港科技大学(广州));Huawei Technologies Co., Ltd.(华为技术有限公司)

Chuanyue Yu, Jiahui Wang, Yuhan Li, Heng Chang, Ge Lan, Qingyun Sun, Jia Li, Jianxin Li, Ziwei Zhang

AI总结:

本文提出SPREAD框架,通过引入查询相关性引导的去噪策略,解决DLMs在RAG框架中生成精度低和语义漂移的问题。

摘要:

扩散语言模型(DLMs)最近在自然语言处理任务中展示了显著的能力。然而,检索增强生成(RAG)的潜力尚未得到充分探索,原因是LLM和DLM解码之间的根本差异。为了填补这一关键空白,我们系统地测试了DLMs在RAG框架中的性能。我们的发现表明,DLMs结合RAG显示出有希望的潜力,但依赖于上下文信息更强,但生成精度有限。我们识别出一个关键的潜在问题:响应语义漂移(RSD),即生成的答案逐渐偏离查询的原始语义,导致低精度内容。我们追溯这个问题到DLMs的去噪策略,这些策略未能在迭代去噪过程中保持与查询的语义对齐。为了解决这个问题,我们提出了语义保持的检索增强扩散(SPREAD),一种新的框架,引入了查询相关性引导的去噪策略。通过主动引导去噪轨迹,SPREAD确保生成保持在查询的语义上,并有效抑制漂移。实验结果表明,SPREAD显著提高了精度,并在RAG框架内有效减轻了生成答案的RSD。

英文摘要:

Diffusion Language Models (DLMs) have recently demonstrated remarkable capabilities in natural language processing tasks. However, the potential of Retrieval-Augmented Generation (RAG), which shows great successes for enhancing large language models (LLMs), has not been well explored, due to the fundamental difference between LLM and DLM decoding. To fill this critical gap, we systematically test the performance of DLMs within the RAG framework. Our findings reveal that DLMs coupled with RAG show promising potentials with stronger dependency on contextual information, but suffer from limited generation precision. We identify a key underlying issue: Response Semantic Drift (RSD), where the generated answer progressively deviates from the query's original semantics, leading to low precision content. We trace this problem to the denoising strategies in DLMs, which fail to maintain semantic alignment with the query throughout the iterative denoising process. To address this, we propose Semantic-Preserving REtrieval-Augmented Diffusion (SPREAD), a novel framework that introduces a query-relevance-guided denoising strategy. By actively guiding the denoising trajectory, SPREAD ensures the generation remains anchored to the query's semantics and effectively suppresses drift. Experimental results demonstrate that SPREAD significantly enhances the precision and effectively mitigates RSD of generated answers within the RAG framework.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑