arXiv 2607.012252026-07-02cs.LGcs.AI

语言-批评模仿学习从次优演示中

Language-Critique Imitation Learning from Suboptimal Demonstrations

Graduate Institute of Communication Engineering, National Taiwan University (NTU)（国立台湾大学电信工程学研究所）；University of Utah（犹他大学）；National Yang Ming Chiao Tung University（国立阳明交通大学）；NTU Artificial Intelligence Center of Research Excellence（国立台湾大学人工智能卓越研究中心）

Chih-Han Yang, Dai-Jie Wu, Yun-Ping Huang, Ping-Chun Hsieh, Kenneth Marino, Shao-Hua Sun

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出语言批评框架，用自然语言作为结构化监督信号从次优演示中学习策略，避免压缩为标量，在连续控制任务上优于基线方法。

摘要：

先前从次优演示中模仿学习的工作通常依赖于压缩的监督信号，如置信度估计、判别器分数或重要性权重。这些标量信号本质上是有限的，因为它们无法明确表达关于任务进展、失败模式或纠正动作的中间推理。我们提出一个语言批评框架，用于从次优演示中模仿学习，该框架利用自然语言作为结构化监督信号，避免了将表达性反馈压缩为标量。我们的方法首先从演示中构建语言标签，明确描述当前进展、识别次优行为并提供细粒度的纠正指导。然后我们引入一个语言批评损失，直接使用这些结构化信号训练策略而不将其降为标量，并将其实例化用于行为克隆和扩散策略，得到LC-BC和LC-DP。我们进一步提供一个理论结果，表明在标准假设下，所提出的目标函数上界了专家性能差距。在实验中，我们在涵盖导航、操作和游戏的各种连续控制任务上评估，我们的方法一致优于强模仿学习和离线强化学习基线。这些结果表明，语言可以作为从次优数据中学习鲁棒策略的一种强大且结构化的监督形式。

英文摘要：

Prior work on imitation learning from suboptimal demonstrations typically relies on compressed supervision signals such as confidence estimates, discriminator scores, or importance weights. These scalar signals are inherently limited, as they cannot explicitly express intermediate reasoning about task progress, failure modes, or corrective actions. We propose a language-critique framework for imitation learning from suboptimal demonstrations that instead leverages natural language as a structured supervision signal, avoiding the collapse of expressive feedback into scalars. Our method first constructs language labels from demonstrations that explicitly describe current progress, identify suboptimal behaviors, and provide fine-grained corrective guidance. We then introduce a language-critique loss that directly trains policies using these structured signals without reducing them to scalars, and instantiate it for both behavior cloning and diffusion policies, yielding LC-BC and LC-DP. We further provide a theoretical result showing that the proposed objective upper-bounds the expert performance gap under standard assumptions. Empirically, we evaluate on diverse continuous control tasks spanning navigation, manipulation, and gameplay, where our methods consistently outperform strong imitation learning and offline reinforcement learning baselines. These results demonstrate that language can serve as a powerful and structured form of supervision for learning robust policies from suboptimal data.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。