arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.19019cs.CLcs.LG
收藏

RAGProbe:一种用于评估 RAG 应用的自动化方法

RAGProbe: An Automated Approach for Evaluating RAG Applications

Applied Artificial Intelligence Institute, Deakin University(迪肯大学应用人工智能研究所)

Shangeetha Sivasothy, Scott Barnett, Stefanus Kurniawan, Zafaryab Rasool, Rajesh Vasa

AI总结:

本文提出 RAGProbe 自动化方法,通过生成问答对变体触发 RAG 流水线失败,并在多个数据集和开源流水线上验证其能显著提高失败检出率,可集成到 CI/CD 中持续监测 RAG 系统质量。

摘要:

检索增强生成(Retrieval Augmented Generation, RAG)在构建生成式 AI 应用时正被越来越多地使用。对这些应用和 RAG 流水线的评估目前大多依赖人工,通过试错过程完成。自动化评估 RAG 流水线需要克服上下文误解、格式错误、特异性不正确以及内容缺失等挑战。因此,先前的工作侧重于利用现有的问答数据集改进评估指标以及增强流水线内部的组件。然而,这些工作并未关注:1)提供一种用于刻画不同类型问答对的模式(schema);2)创建一组用于生成问答对的模板,以支持 RAG 流水线评估的自动化。在本文中,我们提出了一种生成问答对变体的技术,以触发 RAG 流水线中的失败。我们使用 3 个数据集对 5 个开源 RAG 流水线进行了验证。我们的方法揭示了当提示组合多个问题时失败率最高:跨多个文档的问题失败率为 91%,来自单个文档的问题失败率为 78%;这表明开发者需要优先处理这些组合问题。在学术领域数据集中观察到 60% 的失败率,在开放领域数据集中观察到 53% 和 62% 的失败率。我们的自动化方法优于现有的最先进方法,每个数据集的平均失败率提高了 51%。我们的工作提出了一种用于持续监测 RAG 流水线健康状况的自动化方法,该方法可以集成到现有的 CI/CD 流水线中,从而提高质量。

英文摘要:

Retrieval Augmented Generation (RAG) is increasingly being used when building Generative AI applications. Evaluating these applications and RAG pipelines is mostly done manually, via a trial and error process. Automating evaluation of RAG pipelines requires overcoming challenges such as context misunderstanding, wrong format, incorrect specificity, and missing content. Prior works therefore focused on improving evaluation metrics as well as enhancing components within the pipeline using available question and answer datasets. However, they have not focused on 1) providing a schema for capturing different types of question-answer pairs or 2) creating a set of templates for generating question-answer pairs that can support automation of RAG pipeline evaluation. In this paper, we present a technique for generating variations in question-answer pairs to trigger failures in RAG pipelines. We validate 5 open-source RAG pipelines using 3 datasets. Our approach revealed the highest failure rates when prompts combine multiple questions: 91% for questions when spanning multiple documents and 78% for questions from a single document; indicating a need for developers to prioritise handling these combined questions. 60% failure rate was observed in academic domain dataset and 53% and 62% failure rates were observed in open-domain datasets. Our automated approach outperforms the existing state-of-the-art methods, by increasing the failure rate by 51% on average per dataset. Our work presents an automated approach for continuously monitoring the health of RAG pipelines, which can be integrated into existing CI/CD pipelines, allowing for improved quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑