arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.05856cs.SEcs.AI
收藏

工程化检索增强生成系统时的七个失败点

Seven Failure Points When Engineering a Retrieval Augmented Generation System

Applied Artificial Intelligence Institute(应用人工智能研究所);Deakin University(迪肯大学)

Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, Mohamed Abdelrazek

AI总结:

本文通过研究、教育和生物医学三个案例研究,总结出工程化RAG系统时的七个失败点,强调验证仅在运行期可行且鲁棒性需逐步演进。

摘要:

软件工程师越来越多地使用一种称为检索增强生成(RAG)的策略,为应用程序添加语义搜索功能。RAG系统涉及查找与查询语义匹配的文档,然后将文档传递给大型语言模型(LLM)(如ChatGPT),以利用LLM提取正确答案。RAG系统旨在:a)减少LLM产生幻觉响应的问题,b)将来源/引用链接到生成的响应,以及c)消除为文档标注元数据的需要。然而,RAG系统存在信息检索系统固有的局限性以及对LLM的依赖。在本文中,我们通过来自三个不同领域(研究、教育和生物医学)的案例研究,呈现了关于RAG系统失败点的经验报告。我们分享了经验教训,并提出了在设计RAG系统时需要考虑的7个失败点。我们工作得出的两个关键要点是:1)RAG系统的验证仅在运行期间可行,2)RAG系统的鲁棒性是逐步演进的,而非在开始时设计而成。最后,我们为软件工程社区列出了关于RAG系统的潜在研究方向。

英文摘要:

Software engineers are increasingly adding semantic search capabilities to applications using a strategy known as Retrieval Augmented Generation (RAG). A RAG system involves finding documents that semantically match a query and then passing the documents to a large language model (LLM) such as ChatGPT to extract the right answer using an LLM. RAG systems aim to: a) reduce the problem of hallucinated responses from LLMs, b) link sources/references to generated responses, and c) remove the need for annotating documents with meta-data. However, RAG systems suffer from limitations inherent to information retrieval systems and from reliance on LLMs. In this paper, we present an experience report on the failure points of RAG systems from three case studies from separate domains: research, education, and biomedical. We share the lessons learned and present 7 failure points to consider when designing a RAG system. The two key takeaways arising from our work are: 1) validation of a RAG system is only feasible during operation, and 2) the robustness of a RAG system evolves rather than designed in at the start. We conclude with a list of potential research directions on RAG systems for the software engineering community.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑