Software engineers are increasingly adding semantic search capabilities to applications using a strategy known as Retrieval Augmented Generation (RAG). A RAG system involves finding documents that semantically match a query and then passing the documents to a large language model (LLM) such as ChatGPT to extract the right answer using an LLM. RAG systems aim to: a) reduce the problem of hallucinated responses from LLMs, b) link sources/references to generated responses, and c) remove the need for annotating documents with meta-data. However, RAG systems suffer from limitations inherent to information retrieval systems and from reliance on LLMs. In this paper, we present an experience report on the failure points of RAG systems from three case studies from separate domains: research, education, and biomedical. We share the lessons learned and present 7 failure points to consider when designing a RAG system. The two key takeaways arising from our work are: 1) validation of a RAG system is only feasible during operation, and 2) the robustness of a RAG system evolves rather than designed in at the start. We conclude with a list of potential research directions on RAG systems for the software engineering community.
Verified Misguidance: Measuring Structural Citation Failures in Search-Augmented LLMs
Department of Artificial Intelligence, Yonsei University(延世大学人工智能系);Department of Computer Science and Engineering, Konkuk University(Konkuk大学计算机科学与工程系);Incheon International Airport Corporation(仁川国际机场公司);Department of Computer Science and Engineering, Ewha Womans University(成均馆女子大学计算机科学与工程系)
Exploring Multi-Modal Data with Tool-Augmented LLM Agents for Precise Causal Discovery
University of Houston(德克萨斯大学休斯顿分校);NEC Laboratories America(NEC美国实验室);Florida International University(佛罗里达国际大学);North Carolina State University(北卡罗来纳州立大学)
Empowering LLMs for Structure-Based Drug Design via Exploration-Augmented Latent Inference
College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院);Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University(教育部符号计算与知识工程重点实验室);Department of Computer Science, Aalto University(艾尔沃斯大学计算机科学系);College of Artificial Intelligence, Jilin University(吉林大学人工智能学院)
Validating FKG.in: Soundness Assessment in LLM-Augmented Indian Food Knowledge
Ashoka University(阿肖克大学);Mphasis AI and Applied Tech Lab(美斐西斯人工智能与应用技术实验室);Koita Centre for Digital Health, Ashoka University(阿肖克大学科伊塔数字健康中心);Institute for Future Health, UC Irvine(加州大学欧文分校未来健康研究所)
Fabrication After Tool Failure: Tool-Augmented Agents Assert Values Their Tools Did Not Return
The Shishukunj International School(希什昆军国际学校);Haileybury Astana(阿斯塔纳黑利伯瑞学校);UWCSEA East Campus(东南亚世界联合书院东校区);Apta AI(Apta人工智能公司);Spark AI Research(星火人工智能研究院)
Search-Induced Issues in Web-Augmented LLM Code Generation: Detecting and Repairing Error-Inducing Pages
Peking University(北京大学);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);Singapore Management University(新加坡管理大学);Fuzhou University(福州大学)