利用策略蒸馏:使智能体策略与运行时语义对齐
Harness Policy Distillation: Aligning Agent Policies with Runtime Semantics
AI总结:
本研究提出Harness策略蒸馏(HPD),通过恢复运行时语义并联合训练模型,使智能体策略与运行时语义对齐,在TRAJECT-Bench上显著提升了Qwen3-8B的自主任务成功率和运行时合规性。
摘要:
将通用模型部署为智能体,意味着需在特定于产品的运行时harness中运行它,该harness会决定哪些动作可执行、状态如何推进以及循环何时终止。普通轨迹监督仅显示采取了哪个动作,但该动作背后的运行时语义是隐式的。我们提出Harness策略蒸馏(HPD),将这些运行时控制语义与动作一同作为显式训练目标,以对齐harness策略。HPD通过在harness中重放参考轨迹,恢复普通轨迹数据中隐式的运行时语义,并将其表示为每次模型调用边界处的控制文档。这些文档为教师生成动作提供特权信息,而harness验证确保所得目标在运行时契约下可执行。我们在从控制状态特权信息和原生历史中恢复的运行时语义上联合训练共享模型,从而将这些语义内化到模型策略中。部署时,训练后的模型使用原始harness接口,无需控制文档或任何额外补充。在带有Pi harness的TRAJECT-Bench上,采用HPD训练的Qwen3-8B达到22.98%的自主任务成功率和49.23%的运行时合规性,而未适配模型的对应数值为3.09%和19.90%。经harness验证的教师动作显著改进了原生部署策略,而联合控制状态监督进一步提升了任务完成度和运行时合规性。HPD将可执行的运行时语义转化为每次决策边界处学习harness兼容动作的密集监督信号。
英文摘要:
Deploying a general-purpose model as an agent means running it inside a product-specific runtime harness that decides which actions are executable, how state advances, and when the loop may terminate. Ordinary trajectory supervision shows which action was taken but leaves the runtime semantics behind that action implicit. We introduce Harness Policy Distillation (HPD), which makes these runtime control semantics an explicit training target alongside actions to align the harness policy. HPD replays reference trajectories through the harness to recover the runtime semantics that are implicit in ordinary trajectory data, and represents them as control documents at each model-call boundary. These documents provide privileged information for a teacher to generate actions, while harness verification ensures that the resulting targets are executable under the runtime contract. We jointly train a shared model on the runtime semantics recovered from control-state privileged information and native history, thereby internalizing these semantics into the model policy. At deployment, the trained model uses the original harness interface without requiring the control document or any additional supplement. On TRAJECT-Bench with the Pi harness, Qwen3-8B trained with HPD reaches 22.98% autonomous task success and 49.23% runtime compliance, against 3.09% and 19.90% for the unadapted model. Harness-verified teacher actions substantially improve the native deployment policy, while joint control-state supervision provides further gains in task completion and runtime compliance. HPD turns executable runtime semantics into dense supervision for learning harness-compatible actions at each decision boundary.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。