训练智能体达成特定目标或执行期望行为通常通过强化学习实现，尤其是在缺乏专家演示的场景下。然而，通过强化学习支持新的目标或行为需要临时设计合适的奖励函数，这一过程很快会变得难以处理。为解决这一挑战，我们提出面向策略学习的文本感知扩散模型（Text-Aware Diffusion for Policy Learning，TADPoLe），该方法利用预训练的冻结文本条件扩散模型，为文本对齐的策略学习计算密集的零样本奖励信号。我们假设，大规模预训练生成模型编码了丰富的先验知识，这些先验不仅能监督策略以符合文本要求的方式行动，还能使其与从互联网规模训练数据中总结出的“自然性”概念保持一致。实验中我们证明，TADPoLe能够在人形机器人（Humanoid）和狗形机器人（Dog）环境中，学习由自然语言指定的新型目标达成和连续运动行为的策略。这些行为是在无真实奖励或专家演示的零样本条件下习得的，且根据人类评估，其行为在定性上更自然。我们进一步表明，当应用于Meta-World环境中的机器人操作任务时，在无需访问任何领域内演示的情况下，TADPoLe的表现仍具有竞争力。
英文摘要：
Training an agent to achieve particular goals or perform desired behaviors is often accomplished through reinforcement learning, especially in the absence of expert demonstrations. However, supporting novel goals or behaviors through reinforcement learning requires the ad-hoc design of appropriate reward functions, which quickly becomes intractable. To address this challenge, we propose Text-Aware Diffusion for Policy Learning (TADPoLe), which uses a pretrained, frozen text-conditioned diffusion model to compute dense zero-shot reward signals for text-aligned policy learning. We hypothesize that large-scale pretrained generative models encode rich priors that can supervise a policy to behave not only in a text-aligned manner, but also in alignment with a notion of naturalness summarized from internet-scale training data. In our experiments, we demonstrate that TADPoLe is able to learn policies for novel goal-achievement and continuous locomotion behaviors specified by natural language, in both Humanoid and Dog environments. The behaviors are learned zero-shot without ground-truth rewards or expert demonstrations, and are qualitatively more natural according to human evaluation. We further show that TADPoLe performs competitively when applied to robotic manipulation tasks in the Meta-World environment, without having access to any in-domain demonstrations.
Maximum Entropy Reinforcement Learning with Diffusion Policy
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；AiRiA
PPO-UE: Proximal Policy Optimization via Uncertainty-Aware Exploration
Virginia Tech（弗吉尼亚理工大学）；University of Oslo（奥斯陆大学）；DEVCOM Army Research Laboratory（DEVCOM陆军研究实验室）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；University of the District of Columbia（哥伦比亚特区大学）
Distributional Soft Actor-Critic with Diffusion Policy
School of Vehicle and Mobility, Tsinghua University（车辆与移动学院，清华大学）；School of Mechanical Engineering, University of Science and Technology Beijing（机械工程学院，北京科技大学）
DiPOD: Diffusion Policy Optimization without Drifting Apart
University of California, Berkeley（加州大学伯克利分校）；Simons Institute for the Theory of Computing（西蒙斯计算理论研究所）；Department of Electrical Engineering and Computer Sciences, University of California, Berkeley（加州大学伯克利分校电气工程与计算机科学系）
Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning
Hong Kong University of Science and Technology（香港科技大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Southern University of Science and Technology（南方科技大学）
SHAPO: Sharpness-Aware Policy Optimization for Safe Exploration
Université de Montréal（蒙特利尔大学）；Mila - Québec AI Institute（蒙特利尔人工智能研究所）；Université Laval（拉瓦尔大学）；IID - Institut Intelligence et Données（智能与数据研究所）；LawZero；CIFAR AI Chair（CIFAR人工智能主席）
Diffusion Policy through Conditional Proximal Policy Optimization
Southern University of Science and Technology, Shenzhen, China（南方科技大学）；Hong Kong University of Science and Technology, Hongkong, China（香港科技大学）；Zhejiang University-University of Illinois Urbana-Champaign Institute, Haining, China（浙江大学-伊利诺伊大学厄巴纳-香槟分校联合研究所）
wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models
Department of Statistical Science, University College London, United Kingdom（伦敦大学统计科学系）；UCL AI Centre, University College London, United Kingdom（伦敦大学UCL人工智能中心）；Graduate School of AI, Ulsan National Institute of Science and Technology, South Korea（韩国乌山国立科学与技术研究院人工智能研究生院）；Department of Mathematics and Computer Science, Universität Basel, Switzerland（巴塞尔大学数学与计算机科学系）
D2PPO: Diffusion Policy Policy Optimization with Dispersive Loss
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Guangdong Key Laboratory of Big Data Analysis and Processing（广东省大数据分析与处理重点实验室）
Guided Uncertainty-Aware Policy Optimization: Combining Learning and Model-Based Strategies for Sample-Efficient Policy Learning
Stanford University（斯坦福大学）；University of California, Berkeley（加州大学伯克利分校）；Nvidia（英伟达）；University of Toronto（多伦多大学）；University of Sydney（悉尼大学）；University of Washington（华盛顿大学）
