arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.15944cs.SEcs.AIcs.IR
收藏

从PDF开发基于检索增强生成(RAG)的大语言模型系统:一份经验报告

Developing Retrieval Augmented Generation (RAG) based LLM Systems from PDFs: An Experience Report

Tampere University(坦佩雷大学)

Ayman Asad Khan, Md Toufique Hasan, Kai Kristian Kemell, Jussi Rasku, Pekka Abrahamsson

AI总结:

本文以PDF为数据源开发RAG系统,报告端到端流程、技术挑战与解决方案,并对比OpenAI Assistant API与Llama开源模型两种实现路径。

摘要:

本文呈现了一份关于以PDF文档作为主要数据源开发检索增强生成(RAG)系统的经验报告。RAG架构将大语言模型(LLMs)的生成能力与信息检索的精确性相结合。该方法有潜力重新定义我们在生成模型中与结构化及非结构化知识交互并对其进行增强的方式,从而提升响应的透明度、准确性和上下文相关性。论文详细描述了从数据收集、预处理到检索索引和响应生成的端到端流程,重点说明了其中的技术挑战与实际解决方案。我们旨在通过两种不同的方法为开发类似系统的研究人员和实践者提供见解:OpenAI的Assistant API配合GPT系列模型,以及Llama的开源模型。本研究的实际意义在于提升生成式AI系统在领域特定知识和实时信息检索至关重要的各个行业中的可靠性。本工作所使用的Python代码也可在以下地址获取:https://github.com/GPT-Laboratory/RAG-LLM-Development-Guidebook-from-PDFs。

英文摘要:

This paper presents an experience report on the development of Retrieval Augmented Generation (RAG) systems using PDF documents as the primary data source. The RAG architecture combines generative capabilities of Large Language Models (LLMs) with the precision of information retrieval. This approach has the potential to redefine how we interact with and augment both structured and unstructured knowledge in generative models to enhance transparency, accuracy, and contextuality of responses. The paper details the end-to-end pipeline, from data collection, preprocessing, to retrieval indexing and response generation, highlighting technical challenges and practical solutions. We aim to offer insights to researchers and practitioners developing similar systems using two distinct approaches: OpenAI's Assistant API with GPT Series and Llama's open-source models. The practical implications of this research lie in enhancing the reliability of generative AI systems in various sectors where domain-specific knowledge and real-time information retrieval is important. The Python code used in this work is also available at: https://github.com/GPT-Laboratory/RAG-LLM-Development-Guidebook-from-PDFs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑