评估数据库场景下的大语言模型：用于评估其在核心数据库任务中潜力的生命周期基准

中文摘要 AI 辅助

大语言模型（LLMs）正在改变数据库交互范式，从简单的查询翻译工具演变为自主数据库管理员（DBAs）。然而，当前的评估基准不成比例地聚焦于文本转SQL（Text-to-SQL）任务，忽略了从初始模式设计到部署后维护的完整数据库生命周期，这种狭隘的关注无法捕捉现实世界数据库管理所需的多样能力。为填补这一空白，我们推出DBLifeBench——首个在设计、实现、操作、调试和维护五个关键生命周期阶段评估LLMs的基准。此外，为解决模糊自然语言与复杂SQL逻辑之间的认知不匹配问题，我们提出Progressive-Text2SQL，这是一种利用结构化推理图模拟人类迭代问题解决过程的新任务。我们的广泛评估揭示了一个关键见解：通用模型表现出均衡性能，而专门的Text-to-SQL模型在设计和维护等非编码阶段存在“灾难性遗忘”。DBLifeBench为评估和构建真正的全栈数据库智能奠定了基础。

英文摘要

Large Language Models (LLMs) are transforming database interaction paradigms, evolving from simple query translators to autonomous database administrators (DBAs). However, current evaluation benchmarks remain disproportionately fixated on Text-to-SQL tasks, neglecting the holistic Database Lifecycle-from initial schema design to post-deployment maintenance. This narrow focus fails to capture the diverse capabilities required for real-world database management. To bridge this gap, we introduce DBLifeBench, the first benchmark to evaluate LLMs across five critical lifecycle phases: Design, Implementation, Operation, Debugging, and Maintenance. Furthermore, addressing the cognitive mismatch between ambiguous natural language and complex SQL logic, we propose Progressive-Text2SQL, a novel task utilizing structured reasoning graphs to mimic human iterative problem-solving. Our extensive evaluation reveals a critical insight: while general-purpose models demonstrate balanced performance, specialized Text-to-SQL models suffer from ``catastrophic forgetting'' in non-coding phases like design and maintenance. DBLifeBench serves as a foundational step toward evaluating and building true full-stack database intelligence.

发表机构

• East China Normal University（华东师范大学）
• Hasso Plattner Institute/University of Potsdam（哈索·普拉特纳研究所/波茨坦大学）

机构由 AI 辅助整理，请以论文原文为准。