通过增强的 PDF 结构识别革新检索增强生成
Revolutionizing Retrieval-Augmented Generation with Enhanced PDF Structure Recognition
AI总结：
本文针对 PDF 解析损害专业知识 RAG 效果的问题，通过数百个真实文档问题的实证实验证明，具备增强 PDF 结构识别能力的 ChatDOC 可检索更完整片段并显著提升问答质量。
摘要：
随着 Large Language Models（LLMs，大语言模型）的快速发展，Retrieval-Augmented Generation（RAG，检索增强生成）已成为专业知识问答领域的主流方法。当前，主要基础模型公司已经开放 Embedding 和 Chat API 接口，LangChain 等框架也已集成 RAG 流程。看起来 RAG 中的关键模型和步骤都已得到解决，由此产生一个问题：专业知识问答系统现在是否正趋于完美？本文发现，当前主流方法依赖于能够获取高质量文本语料库这一前提。然而，由于专业文档主要以 PDF 格式存储，PDF 解析的低准确性显著影响专业知识问答的效果。我们针对来自相应真实世界专业文档的数百个问题开展了实证 RAG 实验。结果表明，ChatDOC 这一配备全面且精准 PDF 解析器的 RAG 系统能够检索到更准确、更完整的片段，从而给出更好的答案。实证实验显示，ChatDOC 在近 47% 的问题上优于基线，在 38% 的情况下持平，仅在 15% 的情况下表现不足。这表明，我们或许可以通过增强的 PDF 结构识别来革新 RAG。
英文摘要：
With the rapid development of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) has become a predominant method in the field of professional knowledge-based question answering. Presently, major foundation model companies have opened up Embedding and Chat API interfaces, and frameworks like LangChain have already integrated the RAG process. It appears that the key models and steps in RAG have been resolved, leading to the question: are professional knowledge QA systems now approaching perfection? This article discovers that current primary methods depend on the premise of accessing high-quality text corpora. However, since professional documents are mainly stored in PDFs, the low accuracy of PDF parsing significantly impacts the effectiveness of professional knowledge-based QA. We conducted an empirical RAG experiment across hundreds of questions from the corresponding real-world professional documents. The results show that, ChatDOC, a RAG system equipped with a panoptic and pinpoint PDF parser, retrieves more accurate and complete segments, and thus better answers. Empirical experiments show that ChatDOC is superior to baseline on nearly 47% of questions, ties for 38% of cases, and falls short on only 15% of cases. It shows that we may revolutionize RAG with enhanced PDF structure recognition.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
