TIGeR：用大型多模态模型统一文本到图像生成与检索
TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models
AI总结：
本文提出TIGeR框架，用单个大型多模态模型统一文本到图像生成与检索，通过自主决策机制选择最优图像，并构建TIGeR-Bench基准验证其优越性。
摘要：
人类如何有效且高效地获取图像一直是一个长期存在的问题。经典的解决方案是从现有数据库中进行文本到图像的检索；然而，有限的数据库通常缺乏创造力。相比之下，最近在文本到图像生成方面的突破使得生成有吸引力且反事实的视觉内容成为可能，但它在合成知识密集型图像方面面临挑战。在这项工作中，我们重新思考了文本到图像生成与检索之间的关系，提出了一个统一的框架，用单个大型多模态模型（LMM）来完成这两项任务。具体来说，我们首先探索了LMM固有的判别能力，并引入了一种无需训练的、高效的生成式检索方法用于文本到图像检索。随后，我们以自回归方式统一了生成和检索，并提出了一种自主决策机制，在生成的图像和检索到的图像之间选择最匹配的一个作为对文本提示的响应。为了标准化统一文本到图像生成与检索的评估，我们构建了TIGeR-Bench，一个涵盖创意领域和知识密集型领域的基准。在TIGeR-Bench以及两个检索基准（即Flickr30K和MS-COCO）上进行的大量实验证明了我们提出的框架的优越性。
英文摘要：
How humans can effectively and efficiently acquire images has always been a perennial question. A classic solution is text-to-image retrieval from an existing database; however, the limited database typically lacks creativity. By contrast, recent breakthroughs in text-to-image generation have made it possible to produce attractive and counterfactual visual content, but it faces challenges in synthesizing knowledge-intensive images. In this work, we rethink the relationship between text-to-image generation and retrieval, proposing a unified framework for both tasks with one single Large Multimodal Model (LMM). Specifically, we first explore the intrinsic discriminative abilities of LMMs and introduce an efficient generative retrieval method for text-to-image retrieval in a training-free manner. Subsequently, we unify generation and retrieval autoregressively and propose an autonomous decision mechanism to choose the best-matched one between generated and retrieved images as the response to the text prompt. To standardize the evaluation of unified text-to-image generation and retrieval, we construct TIGeR-Bench, a benchmark spanning both creative and knowledge-intensive domains. Extensive experiments on TIGeR-Bench and two retrieval benchmarks, i.e., Flickr30K and MS-COCO, demonstrate the superiority of our proposed framework.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
