arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.00820cs.IRcs.CL
收藏

检索增强生成系统:自动数据集创建、评估与布尔智能体设置

Retrieval Augmented Generation Systems: Automatic Dataset Creation, Evaluation and Boolean Agent Setup

Technische Fakultät University of Bielefeld(比勒费尔德大学工程学院)

Tristan Kenneweg, Philip Kenneweg, Barbara Hammer

AI总结:

针对RAG方案缺乏定量比较的问题,本文构建自动化数据集创建与评估流程,并据此开发可由LLM决定是否检索的布尔智能体RAG,以节省token,同时公开代码与数据集。

摘要:

检索增强生成(RAG)系统在利用领域特定和时间敏感数据增强大语言模型(LLM)输出方面已广受欢迎。最近,相关方向正从简单的RAG设置转向更复杂的RAG形式;在简单设置中,系统会针对每次用户输入查询向量数据库以获取额外信息。然而,目前不同的具体方法主要依靠轶事性证据相互竞争。本文提出一套严谨的数据集创建和评估工作流,用于定量比较不同的RAG策略。我们使用以这种方式创建的数据集来开发和评估一种布尔智能体RAG设置:在该系统中,LLM可以决定是否查询向量数据库,从而在可凭借内部知识回答的问题上节省token。我们在线发布了代码和生成的数据集。

英文摘要:

Retrieval Augmented Generation (RAG) systems have seen huge popularity in augmenting Large-Language Model (LLM) outputs with domain specific and time sensitive data. Very recently a shift is happening from simple RAG setups that query a vector database for additional information with every user input to more sophisticated forms of RAG. However, different concrete approaches compete on mostly anecdotal evidence at the moment. In this paper we present a rigorous dataset creation and evaluation workflow to quantitatively compare different RAG strategies. We use a dataset created this way for the development and evaluation of a boolean agent RAG setup: A system in which a LLM can decide whether to query a vector database or not, thus saving tokens on questions that can be answered with internal knowledge. We publish our code and generated dataset online.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑