UniversalRAG: 在多样模态和粒度的语料库上实现检索增强生成
UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities
AI总结：
本文提出UniversalRAG，一种能够处理多种模态和粒度的检索增强生成框架，通过动态路由机制和多粒度组织，提升跨模态知识检索的有效性，实验表明其在多个模态基准上的优越性。
摘要：
检索增强生成（RAG）通过将外部相关知识与查询绑定，显著提升了事实准确性。然而，现有方法多局限于文本语料，尽管最近有尝试扩展到图像、视频等模态，但通常仅针对单一模态语料。相比之下，现实中的查询所需知识类型多样，单一知识源无法满足。为此，我们引入UniversalRAG，一种any-to-any RAG框架，旨在从异构源中检索和整合多样模态和粒度的知识。具体而言，受强制所有模态进入单一聚合语料的统一表示空间导致模态间隙的观察启发，我们提出模态感知路由，动态识别最合适的模态特定语料并执行针对性检索，并通过理论分析证明其有效性。此外，除模态外，我们对每个模态组织为多个粒度层级，实现针对查询复杂性和范围的精细检索。我们验证UniversalRAG在10个多种模态基准上的性能，显示其优于各种模态特定和统一基线。
英文摘要：
Retrieval-Augmented Generation (RAG) has shown substantial promise in improving factual accuracy by grounding model responses with external knowledge relevant to queries. However, most existing approaches are limited to a text-only corpus, and while recent efforts have extended RAG to other modalities such as images and videos, they typically operate over a single modality-specific corpus. In contrast, real-world queries vary widely in the type of knowledge they require, which a single type of knowledge source cannot address. To address this, we introduce UniversalRAG, an any-to-any RAG framework designed to retrieve and integrate knowledge from heterogeneous sources with diverse modalities and granularities. Specifically, motivated by the observation that forcing all modalities into a unified representation space derived from a single aggregated corpus causes a modality gap, where the retrieval tends to favor items from the same modality as the query, we propose modality-aware routing, which dynamically identifies the most appropriate modality-specific corpus and performs targeted retrieval within it, and further justify its effectiveness with a theoretical analysis. Moreover, beyond modality, we organize each modality into multiple granularity levels, enabling fine-tuned retrieval tailored to the complexity and scope of the query. We validate UniversalRAG on 10 benchmarks of multiple modalities, showing its superiority over various modality-specific and unified baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
