基于化学领域的检索增强生成基准测试
Benchmarking Retrieval-Augmented Generation for Chemistry
AI总结：
本文提出ChemRAG-Bench和Toolkit，用于评估RAG在化学领域的有效性，并展示RAG在化学任务中比直接推理方法有17.4%的性能提升。
摘要：
检索增强生成（RAG）已逐渐成为一种强大的框架，用于增强大型语言模型（LLMs）以获取外部知识，特别是在需要专门化和动态信息的科学领域。尽管其具有潜力，但RAG在化学领域的应用仍处于探索阶段，主要原因是缺乏高质量的领域特定语料库和经过精心编写的评估基准。在本文中，我们介绍了ChemRAG-Bench，一个全面的基准，旨在系统评估RAG在一系列化学相关任务中的有效性。所附的化学语料库整合了异构的知识来源，包括科学文献、PubChem数据库、PubMed摘要、教科书和维基百科条目。此外，我们还提出了ChemRAG-Toolkit，一个模块化且可扩展的RAG工具包，支持五种检索算法和八种LLM。利用ChemRAG-Toolkit，我们证明RAG在直接推理方法上实现了显著的性能提升——平均相对改进为17.4%。我们进一步对检索器架构、语料库选择和检索的段落数量进行了深入分析，最终得出实用的建议，以指导未来RAG系统在化学领域的研究和部署。代码和数据可在https://chemrag.github.io获取。
英文摘要：
Retrieval-augmented generation (RAG) has emerged as a powerful framework for enhancing large language models (LLMs) with external knowledge, particularly in scientific domains that demand specialized and dynamic information. Despite its promise, the application of RAG in the chemistry domain remains underexplored, primarily due to the lack of high-quality, domain-specific corpora and well-curated evaluation benchmarks. In this work, we introduce ChemRAG-Bench, a comprehensive benchmark designed to systematically assess the effectiveness of RAG across a diverse set of chemistry-related tasks. The accompanying chemistry corpus integrates heterogeneous knowledge sources, including scientific literature, the PubChem database, PubMed abstracts, textbooks, and Wikipedia entries. In addition, we present ChemRAG-Toolkit, a modular and extensible RAG toolkit that supports five retrieval algorithms and eight LLMs. Using ChemRAG-Toolkit, we demonstrate that RAG yields a substantial performance gain -- achieving an average relative improvement of 17.4% over direct inference methods. We further conduct in-depth analyses on retriever architectures, corpus selection, and the number of retrieved passages, culminating in practical recommendations to guide future research and deployment of RAG systems in the chemistry domain. The code and data is available at https://chemrag.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
