使用可扩展视觉嵌入检索与后期交互重排序器开发视觉增强问答系统
Developing Visual Augmented Q&A System using Scalable Vision Embedding Retrieval & Late Interaction Re-ranker
AI总结：
本文提出一种多步骤定制方案，利用混合搜索与后期交互重排序器检索最佳匹配页面，并使用多模态大语言模型生成答案，解决了视觉增强问答中检索可扩展性与效率的问题。
摘要：
传统的信息抽取系统在使用纯文本语言模型时面临挑战，因为它没有考虑信息图表（信息的视觉元素），如表格、图表、图像等，而这些通常用于向读者传达复杂信息。多模态大语言模型（MLLM）面临大海捞针问题的挑战，即搜索空间包含更长的上下文长度或大量文档。基于视觉语言模型的后期交互机制在基于检索的视觉增强问答任务中展现出了最先进的性能。然而，将其用于基于RAG的多模态问答仍存在一些挑战。首先，许多流行且广泛采用的向量数据库不支持原生多向量检索。其次，后期交互需要大量计算，这会膨胀空间占用并可能阻碍企业采用。最后，后期交互机制的当前状态未能利用近似邻居搜索索引方法来实现检索过程的大幅加速。本文探索了一种务实的方法，使视觉检索过程在不影响性能质量的前提下具备可扩展性和高效性。我们提出了一种多步骤定制实现方案，利用广泛采用的混合搜索（元数据与嵌入）和最先进的后期交互重排序器来检索最佳匹配页面。最后，将MLLM作为阅读器进行提示，从上下文化的最佳匹配页面生成答案。通过实验，我们观察到所提出的设计具有可扩展性（显著加速）和稳定性（不降低性能质量），因此可以作为企业的生产系统使用。
英文摘要：
Traditional information extraction systems face challenges with text only language models as it does not consider infographics (visual elements of information) such as tables, charts, images etc. often used to convey complex information to readers. Multimodal LLM (MLLM) face challenges of finding needle in the haystack problem i.e., either longer context length or substantial number of documents as search space. Late interaction mechanism over visual language models has shown state of the art performance in retrieval-based vision augmented Q&A tasks. There are yet few challenges using it for RAG based multi-modal Q&A. Firstly, many popular and widely adopted vector databases do not support native multi-vector retrieval. Secondly, late interaction requires computation which inflates space footprint and can hinder enterprise adoption. Lastly, the current state of late interaction mechanism does not leverage the approximate neighbor search indexing methods for large speed ups in retrieval process. This paper explores a pragmatic approach to make vision retrieval process scalable and efficient without compromising on performance quality. We propose multi-step custom implementation utilizing widely adopted hybrid search (metadata & embedding) and state of the art late interaction re-ranker to retrieve best matching pages. Finally, MLLM are prompted as reader to generate answers from contextualized best matching pages. Through experiments, we observe that the proposed design is scalable (significant speed up) and stable (without degrading performance quality), hence can be used as production systems at enterprises.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
