FATHOMS-RAG：评估使用检索增强生成的多模态系统思考与观察的框架
FATHOMS-RAG: A Framework for the Assessment of Thinking and Observation in Multimodal Systems that use Retrieval Augmented Generation
AI总结：
提出一个评估RAG管道的基准，包含93个多模态问题、短语级召回率指标、最近邻嵌入分类器，并对比开源与闭源管道性能，发现闭源管道在正确性和幻觉指标上显著优于开源。
摘要：
检索增强生成（RAG）已成为提高大型语言模型（LLMs）事实准确性的有前景范式。我们引入了一个旨在整体评估RAG管道的基准，评估管道摄取、检索和推理多种模态信息的能力，区别于现有专注于检索等特定方面的基准。我们提出：（1）一个由93个人工创建的问题组成的小型数据集，用于评估管道摄取文本数据、表格、图像以及跨这些模态分布在多个文档中的数据的能力；（2）一个用于正确性的短语级召回率指标；（3）一个最近邻嵌入分类器，用于识别潜在的管道幻觉；（4）对使用开源检索机制构建的2个管道和4个闭源基础模型进行的比较评估；（5）第三方人工评估我们正确性和幻觉指标的对齐情况。我们发现，闭源管道在正确性和幻觉指标上均显著优于开源管道，在依赖多模态和跨文档信息的问题上性能差距更大。对我们指标的人工评估显示，在1-5 Likert量表（5表示“强烈同意”）上，正确性平均一致性为4.62，幻觉检测平均一致性为4.53。
英文摘要：
Retrieval-augmented generation (RAG) has emerged as a promising paradigm for improving factual accuracy in large language models (LLMs). We introduce a benchmark designed to evaluate RAG pipelines as a whole, evaluating a pipeline's ability to ingest, retrieve, and reason about several modalities of information, differentiating it from existing benchmarks that focus on particular aspects such as retrieval. We present (1) a small, human-created dataset of 93 questions designed to evaluate a pipeline's ability to ingest textual data, tables, images, and data spread across these modalities in one or more documents; (2) a phrase-level recall metric for correctness; (3) a nearest-neighbor embedding classifier to identify potential pipeline hallucinations; (4) a comparative evaluation of 2 pipelines built with open-source retrieval mechanisms and 4 closed-source foundation models; and (5) a third-party human evaluation of the alignment of our correctness and hallucination metrics. We find that closed-source pipelines significantly outperform open-source pipelines in both correctness and hallucination metrics, with wider performance gaps in questions relying on multimodal and cross-document information. Human evaluation of our metrics showed average agreement of 4.62 for correctness and 4.53 for hallucination detection on a 1-5 Likert scale (5 indicating "strongly agree").
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
