arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.30105cs.ROcs.AIcs.LG
收藏

非对称在线策略蒸馏学习上下文探索

Asymmetric On-Policy Distillation Learns In-Context Exploration

Paul G. Allen School of Computer Science and Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院);Meta

Sriyash Poddar, Yanda Bao, Jacob Krantz, Matthew Chang, Xavier Puig, Roozbeh Mottaghi, Natasha Jaques, Abhishek Gupta

AI总结:

本研究提出ASTEROID框架,通过非对称在线策略蒸馏,在11类环境中实现高效测试时探索,交互量远低于强化学习基线,且能迁移至真实机器人操作,成功率提升40%。

摘要:

部分可观测下的序列决策难度在于智能体需先探索以推断隐藏的任务相关信息,才能有效行动。通过在线强化学习从头学习此类信息收集行为样本效率低,而离线模仿会在隐藏状态间平均专家动作,无法学习探索。本研究提出ASTEROID,一种简单有效的师生监督学习框架,可学习测试时探索。尽管特权专家无需探索,我们证明基于历史的学生在在线策略上下文训练时可学习探索:学生先在部分可观测下收集在线策略上下文,特权专家提供该上下文条件下的动作标签。通过直接蒸馏学生在线策略历史条件下的专家动作,学生学会系统解决部分可观测问题,即不确定时搜索,隐藏状态明确后利用。这实现了类似贝叶斯后验采样的高效测试时探索,无需承担强化学习不切实际的训练数据成本。在11种不同环境(含离散探索基准、机器人操作、导航及Procgen环境)中,ASTEROID在固定探索预算下回报约高2倍,交互量比强化学习基线少约100倍,且在探索上优于前沿模型。最后,该方法将模拟训练的策略迁移至视觉受限的真实世界机器人操作,展现高效探索,成功率比标准蒸馏提升40%。总体而言,它提供了一种简单却惊人有效的框架来学习高效测试时探索。

英文摘要:

Sequential decision-making under partial observability is difficult because agents must explore to infer hidden, task-relevant information before acting effectively. Learning such information-gathering behaviors tabula rasa via online reinforcement learning is sample-inefficient, whereas offline imitation averages expert actions across hidden states and fails to learn exploration. In this work, we propose ASTEROID, a simple and effective student--teacher supervised learning framework that can learn test-time exploration. Although a privileged expert never needs to explore, we demonstrate that a history-conditioned student can learn to do so when trained with on-policy contexts: the student first collects on-policy context under partial observability, and then imitates the privileged expert conditioned on that context. By distilling expert actions directly conditioned on the student's on-policy history, the student learns to systematically resolve partial observability, i.e., search while uncertain and exploit once the hidden state is revealed. This enables efficient test-time exploration (similar to Bayesian posterior sampling) without incurring the impractical training-data costs of reinforcement learning. Across a diverse range of eleven different environments, including discrete exploration benchmarks, robotic manipulation, navigation, and Procgen environments, ASTEROID achieves ~2x higher returns under fixed exploration budgets, uses ~100x fewer interactions than RL baselines, and outperforms frontier models at exploration. Finally, our method transfers policies trained in simulation to vision-denied real-world robot manipulation, exhibiting efficient exploration and increasing success by +40% over standard distillation. Overall, it provides a simple yet surprisingly effective framework to learn efficient test-time exploration.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑