arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.10885cs.CLcs.DB
收藏

重新思考智能体工作流:在 Text2SQL 任务中评估基于推理的测试时间扩展策略

Rethinking Agentic Workflows: Evaluating Inference-Based Test-Time Scaling Strategies in Text2SQL Tasks

Bosch Research North America, USA(博世美国研究北美洲);Bosch Center for Artificial Intelligence (BCAI)(博世人工智能中心)

Jiajing Guo, Kenil Patel, Jorge Piazentin Ono, Wenbin He, Liu Ren

AI总结:

本文在 BIRD Mini-Dev 基准上评估了六种测试时间扩展策略和四个 LLM,发现 Divide-and-Conquer 提示和少样本演示能持续提升性能,揭示了部署 Text2SQL 系统时准确率、效率和复杂性的权衡。

摘要:

大型语言模型(LLMs)正日益驱动 Text-to-SQL(Text2SQL)系统,使非专业用户能够使用自然语言查询工业数据库。尽管测试时间扩展策略在基于 LLM 的解决方案中展现出潜力,但其在实际应用中的有效性,尤其是结合最新推理模型时,仍不确定。在这项工作中,我们在 BIRD Mini-Dev 基准上对六种轻量级、面向行业的测试时间扩展策略和四个 LLM(包括两个推理模型)进行了基准测试,评估了它们的性能。除了标准准确率指标外,我们还报告了推理延迟和 token 消耗,为实际系统部署提供了相关见解。我们的发现表明,Divide-and-Conquer 提示和少样本演示持续提升了通用型和推理型 LLM 的性能。然而,引入额外的工作流步骤产生了好坏参半的结果,且基础模型的选择起着至关重要的作用。这项工作揭示了部署 Text2SQL 系统时在准确率、效率和复杂性之间的实际权衡。

英文摘要:

Large language models (LLMs) are increasingly powering Text-to-SQL (Text2SQL) systems, enabling non-expert users to query industrial databases using natural language. While test-time scaling strategies have shown promise in LLM-based solutions, their effectiveness in real-world applications, especially with the latest reasoning models, remains uncertain. In this work, we benchmark six lightweight, industry-oriented test-time scaling strategies and four LLMs, including two reasoning models, evaluating their performance on the BIRD Mini-Dev benchmark. Beyond standard accuracy metrics, we also report inference latency and token consumption, providing insights relevant for practical system deployment. Our findings reveal that Divide-and-Conquer prompting and few-shot demonstrations consistently enhance performance for both general-purpose and reasoning-focused LLMs. However, introducing additional workflow steps yields mixed results, and base model selection plays a critical role. This work sheds light on the practical trade-offs between accuracy, efficiency, and complexity when deploying Text2SQL systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑