面向通用检索增强生成的混合模态检索
Towards Mixed-Modal Retrieval for Universal Retrieval-Augmented Generation
AI总结：
本文针对通用RAG中的混合模态检索问题，提出统一检索器Nyx，并构建NyxQA数据集与两阶段训练框架，显著提升视觉语言生成质量。
摘要：
检索增强生成（RAG）已成为一种强大范式，通过从外部语料库检索相关文档来增强大型语言模型（LLM）。然而，现有RAG系统主要关注单模态文本文档，在查询和文档都可能包含混合模态（如文本和图像）的真实场景中往往表现不足。本文解决通用检索增强生成（URAG）挑战，即检索并推理混合模态信息，以改进视觉语言生成。为此，我们提出Nyx，这是一种为URAG场景定制的统一混合模态到混合模态检索器。为缓解真实混合模态数据稀缺问题，我们引入四阶段自动生成与过滤流水线，利用网络文档构建NyxQA，该数据集包含多样化的混合模态问答对，更能反映真实世界信息需求。基于这一高质量数据集，我们为Nyx采用两阶段训练框架：首先在NyxQA以及多种开源检索数据集上进行预训练，随后利用下游视觉语言模型（VLM）的反馈进行监督微调，使检索输出与生成偏好对齐。实验结果表明，Nyx不仅在标准纯文本RAG基准上具有竞争力，还在更通用、更真实的URAG设置中表现出色，显著提升视觉语言任务的生成质量。
英文摘要：
Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for enhancing large language models (LLMs) by retrieving relevant documents from an external corpus. However, existing RAG systems primarily focus on unimodal text documents, and often fall short in real-world scenarios where both queries and documents may contain mixed modalities (such as text and images). In this paper, we address the challenge of Universal Retrieval-Augmented Generation (URAG), which involves retrieving and reasoning over mixed-modal information to improve vision-language generation. To this end, we propose Nyx, a unified mixed-modal to mixed-modal retriever tailored for URAG scenarios. To mitigate the scarcity of realistic mixed-modal data, we introduce a four-stage automated pipeline for generation and filtering, leveraging web documents to construct NyxQA, a dataset comprising diverse mixed-modal question-answer pairs that better reflect real-world information needs. Building on this high-quality dataset, we adopt a two-stage training framework for Nyx: we first perform pre-training on NyxQA along with a variety of open-source retrieval datasets, followed by supervised fine-tuning using feedback from downstream vision-language models (VLMs) to align retrieval outputs with generative preferences. Experimental results demonstrate that Nyx not only performs competitively on standard text-only RAG benchmarks, but also excels in the more general and realistic URAG setting, significantly improving generation quality in vision-language tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
