arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.19232cs.CLcs.AI
收藏

GRAMMAR:面向闭域检索增强语言模型评估的基于事实且模块化的方法论

GRAMMAR: Grounded and Modular Methodology for Assessment of Closed-Domain Retrieval-Augmented Language Model

School of IT, Deakin University(迪肯大学信息技术学院)

Xinzhe Li, Ming Liu, Shang Gao

AI总结:

本文提出GRAMMAR评估框架,通过基于事实的数据生成和模块化评估协议,解决闭域RAG系统因数据私有和真实答案稀缺而难以诊断缺陷模块的问题,并验证其能可靠识别脆弱模块及支持文本形式脆弱性假设检验。

摘要:

检索增强生成(Retrieval-Augmented Generation,RAG)系统在各行各业中被广泛用于查询闭域和内部知识库。然而,由于闭域数据的私有性以及缺乏具有可验证真实答案的查询,评估这些系统面临重大挑战。此外,目前缺乏分析方法来诊断存在问题的模块并识别失败类型,例如由知识缺陷或鲁棒性问题引起的失败。为应对这些挑战,我们提出GRAMMAR(GRounded And Modular Methodology for Assessment of RAG,即基于事实且模块化的RAG评估方法论),这是一个评估框架,包含一个基于事实的数据生成过程和一个能够有效定位缺陷模块的评估协议。我们的验证实验表明,GRAMMAR为识别脆弱模块提供了一种可靠的方法,并支持对文本形式脆弱性进行假设检验。该框架附带的开源工具可在我们的GitHub仓库中获取(见 https://github.com/xinzhel/grammar),便于轻松复现我们的结果,并支持在闭域环境中进行可靠且模块化的评估。

英文摘要:

Retrieval-Augmented Generation (RAG) systems are widely used across various industries for querying closed-domain and in-house knowledge bases. However, evaluating these systems presents significant challenges due to the private nature of closed-domain data and a scarcity of queries with verifiable ground truths. Moreover, there is a lack of analytical methods to diagnose problematic modules and identify types of failure, such as those caused by knowledge deficits or issues with robustness. To address these challenges, we introduce GRAMMAR (GRounded And Modular Methodology for Assessment of RAG), an evaluation framework comprising a grounded data generation process and an evaluation protocol that effectively pinpoints defective modules. Our validation experiments reveal that GRAMMAR provides a reliable approach for identifying vulnerable modules and supports hypothesis testing for textual form vulnerabilities. An open-source tool accompanying this framework is available in our GitHub repository (see https://github.com/xinzhel/grammar), allowing for easy reproduction of our results and enabling reliable and modular evaluation in closed-domain settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑