arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.04719cs.CL
收藏

每一步都重要:面向工具集成文本到SQL的步骤级信用分配

Every Step Counts: Step-Level Credit Assignment for Tool-Integrated Text-to-SQL

Institute of Computer Science and Technology, Soochow University, China(苏州大学计算机科学与技术学院);Ant Digital Technologies, Ant Group(蚂蚁集团数字技术部);School of Management, University of Science and Technology of China(中国科学技术大学管理学院)

Yaxun Dai, Baolin Sun, Junying Wang, Pengfei Wang, Yingqi Gao, Xuemei Dong, Mengdie Chu, Xiang Qi, Pingfu Chao

AI总结:

本文提出FineStep框架,通过独立过程奖励设计和步骤级信用分配机制,解决文本到SQL中粗粒度监督导致的信用分配问题,提升模型效率和泛化能力。

摘要:

工具集成的文本到SQL解析已发展为有前景的范式,将SQL生成视为交织工具执行的序列决策过程。然而,现有强化学习方法主要依赖粗粒度结果监督,导致根本性的信用分配问题:模型对产生正确答案的任何轨迹给予相同奖励,即使中间步骤冗余、低效或错误。因此,模型被鼓励探索次优推理空间,限制了效率和泛化能力。为解决此问题,我们提出FineStep,一种用于工具增强文本到SQL的步骤级信用分配新框架。首先,我们引入独立过程奖励设计以缓解结果监督的信号稀疏性。其次,我们提出步骤级信用分配机制以精确量化每个推理步骤的价值。最后,我们开发基于步骤级优势的策略优化方法以实现高效更新。在BIRD基准测试中,广泛实验表明FineStep实现了最先进的性能,并减少了冗余工具交互,在4B规模下比GRPO平均提高3.25%的EX。

英文摘要:

Tool-integrated Text-to-SQL parsing has emerged as a promising paradigm, framing SQL generation as a sequential decision-making process interleaved with tool execution. However, existing reinforcement learning approaches mainly rely on coarse-grained outcome supervision, resulting in a fundamental credit assignment problem: models receive the same reward for any trajectory that yields the correct answer, even when intermediate steps are redundant, inefficient, or erroneous. Consequently, models are encouraged to explore suboptimal reasoning spaces, limiting both efficiency and generalization. To address this problem, we propose FineStep, a novel framework for step-level credit assignment in tool-augmented Text-to-SQL. First, we introduce a reward design with independent process rewards to alleviate the signal sparsity of outcome supervision. Next, we present a step-level credit assignment mechanism to precisely quantify the value of each reasoning step. Finally, we develop a policy optimization method based on step-level advantages for efficient updates. Extensive experiments on BIRD benchmarks show that FineStep achieves state-of-the-art performance and reduces redundant tool interactions, with a 3.25% average EX gain over GRPO at the 4B scale.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑