arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.04475cs.DBcs.AI
收藏

用大语言模型革新数据库问答：综合基准与评估

Revolutionizing Database Q&A with Large Language Models: Comprehensive Benchmark and Evaluation

Xiamen University（厦门大学）；Tsinghua University（清华大学）；Huawei（华为）

Yihang Zheng, Bo Li, Zhenghao Lin, Yi Luo, Xuanhe Zhou, Chen Lin, Jinsong Su, Guoliang Li, Shifu Li

备注：
12 pages
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出首个面向LLM数据库问答的综合基准DQABench与模块化测试平台DQATestbed，基于超20万中英问答对评估九个问答机器人及RAG、TIG等组件。

摘要：

大语言模型（LLM）的发展已经革新了包括数据库领域在内的多个行业的问答（QA）。然而，目前仍缺乏一个综合基准来评估不同LLM及其模块化组件在数据库问答中的能力。为此，我们提出了DQABench，这是首个面向LLM的综合数据库问答基准。DQABench采用一种创新的基于LLM的方法，自动化完成评估数据集的生成、清洗和重写，最终分别形成超过200,000个英文和中文问答对。这些问答对覆盖了从手册、在线社区和数据库实例中提取的广泛数据库相关知识。这种纳入还使得能够额外评估LLM在数据库问答任务中的检索增强生成（RAG）和工具调用生成（TIG）能力。此外，我们提出了一个综合的基于LLM的数据库问答测试平台DQATestbed。该测试平台具有高度模块化和可扩展性，包含问题分类路由（QCR）、RAG、TIG和提示模板工程（PTE）等基础与高级组件。此外，DQABench提供了综合评估流程，在标准化评估过程中计算多种指标，以确保评估的准确性和公平性。我们使用DQABench在所提出的测试平台下全面评估数据库问答能力。评估揭示了以下发现：（i）九个基于LLM的问答机器人的优势与局限；（ii）各类服务组件（例如QCR、RAG、TIG）的性能影响和潜在改进。我们的基准和发现将指导未来基于LLM的数据库问答研究的发展。

英文摘要：

The development of Large Language Models (LLMs) has revolutionized QA across various industries, including the database domain. However, there is still a lack of a comprehensive benchmark to evaluate the capabilities of different LLMs and their modular components in database QA. To this end, we introduce DQABench, the first comprehensive database QA benchmark for LLMs. DQABench features an innovative LLM-based method to automate the generation, cleaning, and rewriting of evaluation dataset, resulting in over 200,000 QA pairs in English and Chinese, separately. These QA pairs cover a wide range of database-related knowledge extracted from manuals, online communities, and database instances. This inclusion allows for an additional assessment of LLMs' Retrieval-Augmented Generation (RAG) and Tool Invocation Generation (TIG) capabilities in the database QA task. Furthermore, we propose a comprehensive LLM-based database QA testbed DQATestbed. This testbed is highly modular and scalable, with basic and advanced components such as Question Classification Routing (QCR), RAG, TIG, and Prompt Template Engineering (PTE). Moreover, DQABench provides a comprehensive evaluation pipeline that computes various metrics throughout a standardized evaluation process to ensure the accuracy and fairness of the evaluation. We use DQABench to evaluate the database QA capabilities under the proposed testbed comprehensively. The evaluation reveals findings like (i) the strengths and limitations of nine LLM-based QA bots and (ii) the performance impact and potential improvements of various service components (e.g., QCR, RAG, TIG). Our benchmark and findings will guide the future development of LLM-based database QA research.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑