arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.26474cs.LGcs.AI
收藏

将RL诱导的工具使用定位到单个交叉编码器特征

Localizing RL-Induced Tool Use to a Single Crosscoder Feature

Andrii Shportko, Shubham Bhokare, Ahmed Zeyad A Alzahrani, Bowen Cheng, Gustavo Mercier, Jessica Hullman

AI总结:

提出专用特征交叉编码器(DFC)隔离强化学习引入的紧凑特征集,通过编码-解码重建提升工具调用正确率31.1个百分点,并实现能力溢出至冻结基模型。

摘要:

通过强化学习微调重塑了语言模型的内部表示,使其能够实现工具使用等代理行为,但这些变化的机制基础仍知之甚少。虽然RL显著改善了结构化工具调用的生成,但尚不清楚哪些特征出现、哪些被保留,以及识别出的特征是否可用于无重训的行为控制。在这项工作中,我们展示了$\textit{专用特征交叉编码器(DFC)}$隔离了一组紧凑的RL特定特征,这些特征在$\texttt{Qwen2.5-3B}$中介导了工具调用能力。在$48$个交叉编码器超参数扫描中,编码-解码重建将RL模型的工具正确性提高了$+31.1 \pm {9.7}$个百分点,并被动地将工具调用能力转移给冻结的基模型,提高了$+6.8 \pm 5.0$个百分点,我们称之为$\textit{能力溢出}$。我们的发现表明,DFC分区将RL引入的能力集中到一个最小、可操控的特征集中,从而实现了对代理LLM的运行时行为控制。

英文摘要:

Fine-tuning through RL reshapes the internal representations of language models to enable agentic behaviors such as tool use, yet the mechanistic basis of these changes remains poorly understood. While RL substantially improves structured tool-call generation, it is unclear which features emerge, which are preserved, and whether identified features can be leveraged for retraining-free behavioral control. In this work, we show that $\textit{Dedicated Feature Crosscoders (DFC)}$ isolate a compact set of RL-specific features that mediate tool-calling capability in $\texttt{Qwen2.5-3B}$. Across a $48$-crosscoder hyperparameter sweep, encode-decode reconstruction improves the RL model's tool correctness by $+31.1 \pm {9.7}$ pp and passively transfers tool-calling ability to the frozen base model by $+6.8 \pm 5.0$ pp which we call a $\textit{capability spillover}$. Our findings show that DFC partitioning concentrates RL-introduced capability into a minimal, steerable feature set that enables runtime behavioral control of agentic LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑