arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.08596cs.AIcs.HC
收藏

将LLM推理蒸馏为可解释的策略树用于人机协作

Distilling LLM Reasoning into an Interpretable Policy Tree for Human-AI Collaboration

Sun Yat-sen University(中山大学)

Beiwen Zhang, Yongheng Liang, Guowei Zou, Haitao Wang, Hejun Wu

AI总结:

提出Co-pi-tree方法,通过将大语言模型推理蒸馏为可执行策略树,在Overcooked-AI中平均奖励提升35.4%,同时减少77.7%的LLM查询和97.1%的测试延迟。

摘要:

构建高效可靠的策略以辅助人类是人机协作中不可或缺的。现有方法主要遵循两条工作路线。大多数先前工作依赖多智能体强化学习(MARL)来学习黑盒策略,这限制了可解释性并引发安全问题。近期方法在每个决策步骤查询大语言模型(LLM),导致响应缓慢和推理成本高昂。我们提出协作策略树(Co-pi-tree),一种闭环方法,学习一个可执行的策略树,该树由伙伴行为预测树和智能体动作选择树组成。Co-pi-tree通过将LLM推理蒸馏为策略树代码来构建策略。然后通过伙伴交互评估策略,获取反馈,并使用自然语言总结交互反馈以改进有问题的分支。在Overcooked-AI中的实验表明,Co-pi-tree将平均奖励比基线平均值提高35.4%,同时将LLM查询次数减少77.7%,测试时延迟减少97.1%。项目页面:https://beiwenzhang.github.io/Co-pi-tree/

英文摘要:

Constructing efficient and reliable policies to assist humans is indispensable for human-AI collaboration. Existing methods mainly follow two lines of work. Most prior work relies on multi-agent reinforcement learning (MARL) to learn black-box policies, which limits interpretability and raises safety concerns. Recent methods query large language models (LLMs) at each decision step, causing slow responses and high inference costs. We propose Collaboration Policy Tree (Co-pi-tree), a closed-loop method that learns an executable policy tree consisting of a partner-behavior prediction tree and an agent-action selection tree. Co-pi-tree constructs a policy by distilling LLM reasoning into policy tree code. It then evaluates the policy through partner interaction, obtains feedback, and uses natural language to summarize the interaction feedback to improve problematic branches. Experiments in Overcooked-AI show that Co-pi-tree improves average reward by 35.4% over the baseline average, while reducing the number of LLM queries by 77.7% and test-time latency by 97.1%. Project page: https://beiwenzhang.github.io/Co-pi-tree/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑