RS-Diffuser: 基于分布价值引导的风险敏感扩散规划
RS-Diffuser: Risk-Sensitive Diffusion Planning with Distributional Value Guidance
AI总结：
提出RS-Diffuser，一种结合扩散轨迹生成与分布价值评论家的风险敏感离线规划框架，通过尾部感知目标引导去噪过程，实现灵活的风险偏好行为，在风险敏感D4RL和机器人导航基准上达到最优性能。
摘要：
离线强化学习使得从固定数据集中进行策略学习成为可能，无需额外的环境交互，这对于在线探索成本高昂或不安全的安全关键应用具有吸引力。基于扩散的决策方法最近通过建模丰富的多模态轨迹分布，在离线强化学习中取得了强劲性能。然而，现有的扩散规划器通常是风险中性的，因此可能忽略在实际部署中至关重要的罕见但灾难性的结果。在这项工作中，我们提出了RS-Diffuser，一种风险敏感的离线扩散规划框架，它将基于扩散的轨迹生成与分布价值评论家相结合。RS-Diffuser学习一个关于未来状态轨迹的扩散规划器、一个用于动作解码的独立逆动力学模型，以及一个通过分位数回归估计候选计划完整回报分布的蒙特卡洛分布评论家。在采样时，我们将风险敏感引导信号融入去噪过程，使用从尾部感知目标（如条件风险价值）计算的梯度来引导生成朝向期望的风险特征。因此，一个训练好的模型可以通过仅改变推理时的风险参数，灵活地产生风险规避、风险中性或风险寻求行为。在风险敏感的D4RL和危险机器人导航基准上的大量实验表明，RS-Diffuser达到了最先进的性能，在提高整体回报和最坏情况鲁棒性的同时减少了安全违规。
英文摘要：
Offline reinforcement learning enables policy learning from fixed datasets without additional environment interaction, making it appealing for safety-critical applications where online exploration is costly or unsafe. Diffusion-based decision-making methods have recently achieved strong performance in offline RL by modeling rich, multimodal trajectory distributions. However, existing diffusion planners are typically risk-neutral and therefore may overlook rare but catastrophic outcomes that are crucial in real-world deployment. In this work, we propose RS-Diffuser, a risk-sensitive offline diffusion planning framework that combines diffusion-based trajectory generation with distributional value critics. RS-Diffuser learns a diffusion planner over future state trajectories, a separate inverse dynamics model for action decoding, and a Monte Carlo distributional critic that estimates the full return distribution of candidate plans through quantile regression. At sampling time, we incorporate a risk-sensitive guidance signal into the denoising process, using gradients computed from tail-aware objectives such as Conditional Value at Risk to steer generation toward desired risk profiles. As a result, a single trained model can flexibly produce risk-averse, risk-neutral, or risk-seeking behaviors by changing only the inference-time risk parameter. Extensive experiments on risk-sensitive D4RL and risky robot navigation benchmarks demonstrate that RS-Diffuser achieves state-of-the-art performance, improving both overall return and worst-case robustness while reducing safety violations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
