跨模态 RAG：子维度文本到图像检索增强生成
Cross-modal RAG: Sub-dimensional Text-to-Image Retrieval-Augmented Generation
AI总结：
针对复杂文生图查询难以由单张检索图像覆盖的问题，提出将查询与图像分解为子维度并混合稀疏、稠密检索的 Cross-modal RAG，在多个基准上提升检索与生成效果。
摘要：
文本到图像生成日益需要获取预训练模型无法完全捕捉的特定领域、细粒度且快速演变的知识，因此有必要整合检索方法。现有的检索增强生成（RAG）方法试图通过检索全局相关图像来解决这一问题，但当没有任何单一图像包含复杂用户查询中的全部所需元素时，它们就会失效。我们提出 Cross-modal RAG，这是一种将查询和图像都分解为子维度组件的新颖框架，可实现子查询感知的检索与生成。我们的方法引入混合检索策略——将子维度稀疏检索器与稠密检索器相结合——以识别一个 Pareto 最优图像集合，其中每张图像都贡献查询的互补方面。在生成过程中，多模态大型语言模型受到引导，选择性地以与特定子查询对齐的相关视觉特征为条件，从而确保子查询感知的图像合成。在 MS-COCO、Flickr30K、WikiArt、CUB 和 ImageNet-LT 上的大量实验表明，Cross-modal RAG 在检索方面显著优于现有基线，并进一步提升生成质量，同时保持高效率。
英文摘要：
Text-to-image generation increasingly demands access to domain-specific, fine-grained, and rapidly evolving knowledge that pretrained models cannot fully capture, necessitating the integration of retrieval methods. Existing Retrieval-Augmented Generation (RAG) methods attempt to address this by retrieving globally relevant images, but they fail when no single image contains all desired elements from a complex user query. We propose Cross-modal RAG, a novel framework that decomposes both queries and images into sub-dimensional components, enabling subquery-aware retrieval and generation. Our method introduces a hybrid retrieval strategy - combining a sub-dimensional sparse retriever with a dense retriever - to identify a Pareto-optimal set of images, each contributing complementary aspects of the query. During generation, a multimodal large language model is guided to selectively condition on relevant visual features aligned to specific subqueries, ensuring subquery-aware image synthesis. Extensive experiments on MS-COCO, Flickr30K, WikiArt, CUB, and ImageNet-LT demonstrate that Cross-modal RAG significantly outperforms existing baselines in the retrieval and further contributes to generation quality, while maintaining high efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
