arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.18459cs.LGcs.CL
收藏

基于双向KL约束的小型语言模型的教师引导工具学习

Teacher-Guided Tool Learning for Small Language Models via Bidirectional KL Constraints

Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Security Capability Center(360安全能力中心);Nanjing University(南京大学);Tongji University(同济大学);Shanghai Jiao Tong University(上海交通大学)

Haoxuan Gao, Qin-Wen Luo, Xiang Chen, Yu Tian, Zequn Sun, Xiaofeng Cao, Jiangchao Yao, Sheng-Jun Huang

AI总结:

针对小型语言模型工具学习的挑战,提出SeTPO方法,结合双向KL约束,在BFCL v4和API Bank数据集上实现优于基线的性能。

摘要:

外部工具可扩展小型语言模型(SLMs)的能力,使其能在资源受限环境中部署。但SLMs能力有限,仅通过强化学习难以发现有效工具使用行为,而直接教师监督存在师生行为分布不匹配问题,可能传播错误偏好。为应对这些挑战,本文提出选择性教师引导策略优化(SeTPO),该方法通过选择性教师引导与持续演示监督协调学生探索。具体而言,SeTPO将反向KL推导的 token 级教师反馈融入低奖励轨迹的策略优势,同时辅助正向KL监督在整个训练过程中保留教师演示的行为。在理想双向KL目标下,本文从理论上证明最优策略的期望奖励不低于教师的期望奖励,且保留了教师概率质量的一定比例。在BFCL v4和API Bank上开展的大量实验表明,SeTPO在所有三种学生规模下,相较于监督微调、强化学习和蒸馏基线,均实现了更优的整体性能。

英文摘要:

External tools can extend the capabilities of small language models (SLMs), enabling their deployment in resource constrained environments. However, their limited capabilities make it difficult for SLMs to discover effective tool use behaviors through reinforcement learning alone, while direct teacher supervision suffers from teacher student behavioral distribution mismatch and may propagate erroneous preferences. To address these challenges, we propose Selective Teacher guided Policy Optimization (SeTPO), which coordinates student exploration with selective teacher guidance and persistent demonstration supervision. Specifically, SeTPO incorporates reverse KL derived token level teacher feedback into policy advantages on low reward trajectories, while auxiliary forward KL supervision preserves teacher demonstrated behaviors throughout training. Under an idealized bidirectional KL objective, we theoretically establish that the optimal policy achieves expected reward no lower than the teacher's while retaining a guaranteed fraction of the teacher's probability mass. Extensive experiments on BFCL v4 and API Bank demonstrate that SeTPO achieves superior overall performance over supervised fine tuning, reinforcement learning, and distillation baselines across all three student scales.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑