arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.16627cs.ROcs.AIcs.HCcs.LG
收藏

通过人类交互进行推理时策略引导

Inference-Time Policy Steering through Human Interactions

MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);NVIDIA(英伟达公司)

Yanwei Wang, Lirui Wang, Yilun Du, Balakumar Sundaralingam, Xuning Yang, Yu-Wei Chao, Claudia Perez-D'Arpino, Dieter Fox, Julie Shah

AI总结:

提出推理时策略引导(ITPS)框架,利用人类交互偏置生成采样,避免分布外错误,在三个基准上以随机采样配合扩散策略取得对齐与分布偏移的最佳权衡。

摘要:

由人类演示训练的生成式策略能够自主完成多模态、长时域任务。然而,在推理过程中,人类通常被排除在策略执行循环之外,限制了在多预测中引导预训练策略朝向特定子目标或轨迹形状的能力。天真的(naive)人类干预可能无意中加剧分布偏移,导致约束违反或执行失败。为了在不引发分布外错误的情况下更好地将策略输出与人类意图对齐,我们提出了一种推理时策略引导(Inference-Time Policy Steering, ITPS)框架,该框架利用人类交互来偏置生成式采样过程,而不是在交互数据上对策略进行微调。我们在三个模拟和真实世界基准上评估了ITPS,测试了三种人类交互形式及相应的对齐距离度量。在六种采样策略中,我们提出的带扩散策略的随机采样在对齐与分布偏移之间取得了最佳权衡。视频可在 https://yanweiw.github.io/itps/ 获取。

英文摘要:

Generative policies trained with human demonstrations can autonomously accomplish multimodal, long-horizon tasks. However, during inference, humans are often removed from the policy execution loop, limiting the ability to guide a pre-trained policy towards a specific sub-goal or trajectory shape among multiple predictions. Naive human intervention may inadvertently exacerbate distribution shift, leading to constraint violations or execution failures. To better align policy output with human intent without inducing out-of-distribution errors, we propose an Inference-Time Policy Steering (ITPS) framework that leverages human interactions to bias the generative sampling process, rather than fine-tuning the policy on interaction data. We evaluate ITPS across three simulated and real-world benchmarks, testing three forms of human interaction and associated alignment distance metrics. Among six sampling strategies, our proposed stochastic sampling with diffusion policy achieves the best trade-off between alignment and distribution shift. Videos are available at https://yanweiw.github.io/itps/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑