超越文本：面向工业应用的多模态输入RAG优化
Beyond Text: Optimizing RAG with Multimodal Inputs for Industrial Applications
AI总结：
针对工业领域RAG系统，研究通过对比两种图像处理策略和两种大模型，验证多模态输入可提升性能，且图像文本摘要方法比多模态嵌入更具潜力。
摘要：
大语言模型（LLM）在问答任务中展现出出色能力，但缺乏领域特定知识且易产生幻觉。检索增强生成（RAG）是应对这些挑战的一种方法，而多模态模型正成为可同时处理文本和图像的极具潜力的AI助手。本文开展了一系列实验，旨在确定如何将多模态模型最佳整合到工业领域的RAG系统中。实验目的是验证工业领域文档中加入图像与文本是否能提升RAG性能，并找到这类多模态RAG系统的最优配置。实验包含两种图像处理与检索方法，以及两种用于答案生成的LLM（GPT4-Vision和LLaVA）。这些图像处理策略涉及多模态嵌入的使用，以及从图像生成文本摘要。我们采用LLM-as-a-Judge方法评估实验。结果表明，多模态RAG性能优于单模态RAG设置，尽管图像检索比文本检索难度更大。此外，与使用多模态嵌入相比，利用图像生成的文本摘要更具发展前景，为未来研究提供了更多方向。
英文摘要：
Large Language Models (LLMs) have demonstrated impressive capabilities in answering questions, but they lack domain-specific knowledge and are prone to hallucinations. Retrieval Augmented Generation (RAG) is one approach to address these challenges, while multimodal models are emerging as promising AI assistants for processing both text and images. In this paper we describe a series of experiments aimed at determining how to best integrate multimodal models into RAG systems for the industrial domain. The purpose of the experiments is to determine whether including images alongside text from documents within the industrial domain increases RAG performance and to find the optimal configuration for such a multimodal RAG system. Our experiments include two approaches for image processing and retrieval, as well as two LLMs (GPT4-Vision and LLaVA) for answer synthesis. These image processing strategies involve the use of multimodal embeddings and the generation of textual summaries from images. We evaluate our experiments with an LLM-as-a-Judge approach. Our results reveal that multimodal RAG can outperform single-modality RAG settings, although image retrieval poses a greater challenge than text retrieval. Additionally, leveraging textual summaries from images presents a more promising approach compared to the use of multimodal embeddings, providing more opportunities for future advancements.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
