arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.06397cs.IRcs.LG
收藏

通过强化学习编译扩散实现高效、属性对齐的扇出检索

Efficient, Property-Aligned Fan-Out Retrieval via RL-Compiled Diffusion

University of Illinois Urbana Champaign(伊利诺伊大学厄巴纳-香槟分校)

Pengcheng Jiang, Judith Yue Li, Moonkyung Ryu, R. Lily Hu, Kun Su, Zhong Yi Wan, Liam Hebert, Hao Peng, Jiawei Han, Dima Kuzmin, Craig Boutilier

AI总结:

R4T通过强化学习编译扩散模型,实现高效且属性对齐的扇出检索,提升检索质量并显著降低推理延迟。

摘要:

许多现代检索问题都是集合值问题:给定广泛意图,系统必须返回一组结果,以优化更高阶属性(例如多样性、覆盖性、互补性、连贯性)的同时保持与固定数据库的关联。集合值目标通常是非可分解的,并且无法通过现有的监督(查询,内容)数据集来捕捉,这些数据集只优先考虑单个检索。因此,扇出检索常被用来生成多样化的子查询以检索项目集。虽然强化学习(RL)可以通过交互来优化集合级目标,但在推理时部署一个经过RL调优的LLM进行扇出检索是成本高昂的。相反,基于扩散的生成检索能够在嵌入空间中实现高效的单次扇出,但需要与目标对齐的训练目标。为了解决这些问题,我们提出了R4T(Retrieve-for-Train),它在三步过程中使用RL一次作为目标转换器:(i)使用复合集合级奖励训练扇出LLM,(ii)合成目标一致的训练对,(iii)训练一个轻量级扩散检索器以建模集合值输出的条件分布。在包含精心挑选项目集的大规模时尚和音乐基准测试中,我们展示了R4T在保持检索质量的同时,将查询时间的扇出延迟降低了数量级。

英文摘要:

Many modern retrieval problems are set-valued: given a broad intent, the system must return a collection of results that optimizes higher-order properties (e.g., diversity, coverage, complementarity, coherence) while remaining grounded with respect to a fixed database. Set-valued objectives are typically non-decomposable and are not captured by existing supervised (query, content) datasets which only prioritize top-1 retrieval. Consequently, fan-out retrieval is often employed to generate diverse subqueries to retrieve item sets. While reinforcement learning (RL) can optimize set-level objectives via interaction, deploying an RL-tuned LLM for fan-out retrieval is prohibitively expensive at inference time. Conversely, diffusion-based generative retrieval enables efficient single-pass fan-out in embedding space, but requires objective-aligned training targets. To address these issues, we propose R4T (Retrieve-for-Train), which uses RL once as an objective transducer in a three-step process: (i) train a fan-out LLM with composite set-level rewards, (ii) synthesize objective-consistent training pairs, and (iii) train a lightweight diffusion retriever to model the conditional distribution of set-valued outputs. Across large-scale fashion and music benchmarks consisting of curated item sets, we show that R4T improves retrieval quality relative to strong baselines while reducing query-time fan-out latency by an order of magnitude.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑