RAMAC: 多模态风险感知离线强化学习及行为正则化的作用
RAMAC: Multimodal Risk-Aware Offline Reinforcement Learning and the Role of Behavior Regularization
AI总结：
提出RAMAC框架，结合分布性评论家与生成式演员（如扩散模型），通过条件风险价值与行为克隆的复合目标实现离线强化学习中的风险敏感学习，抑制分布外动作并提升CVaR。
摘要：
在安全关键领域中，当在线数据收集不可行时，离线强化学习（RL）只有在策略能够实现高回报且避免灾难性的下尾风险时才具有吸引力。先前关于风险厌恶离线RL的工作通过（i）基于值/模型的悲观主义或（ii）限制策略类以限制表达能力来实现安全性，而扩散/流式表达性生成策略主要在中性风险设置中使用。我们引入了\textbf{风险感知多模态演员-评论家（RAMAC）}，一个简单、模块化、无模型的框架，它将表达性生成演员（例如扩散/流）与分布性评论家相结合，并优化一个结合条件风险价值（CVaR）与行为克隆（BC）的复合目标，从而在复杂的多模态场景中实现风险敏感学习。由于分布外（OOD）动作是离线RL中灾难性失败的主要驱动因素，我们进一步提供了一个目标层面的分析，表明通过BC控制行为发散可以抑制OOD动作并稳定CVaR。使用扩散演员实例化RAMAC，我们在二维风险赌博机上展示了这些见解，并在Stochastic-D4RL上进行了评估，观察到在保持高回报的同时，$\mathrm{CVaR}_{0.1}$的一致提升。代码和实验结果可在\href{this https URL}{项目网站}上获取。
英文摘要：
In safety-critical domains where online data collection is infeasible, offline reinforcement learning (RL) is attractive only if policies achieve high returns without catastrophic lower-tail risk. Prior work on risk-averse offline RL achieves safety at the cost of either (i) value/model-based pessimism or (ii) restricted policy classes that limit expressiveness, whereas diffusion/flow-based expressive generative policies have largely been used in risk-neutral settings. We introduce \textbf{Risk-Aware Multimodal Actor-Critic (RAMAC)}, a simple, modular, model-free framework that couples an expressive generative actor (e.g., diffusion/flow) with a distributional critic and optimizes a composite objective that combines Conditional Value-at-Risk (CVaR) with behavioral cloning (BC), enabling risk-sensitive learning in complex multimodal scenarios. Since out-of-distribution (OOD) actions are a major driver of catastrophic failures in offline RL, we further provide an objective-level analysis showing that controlling behavior divergence via BC suppresses OOD actions and stabilizes CVaR. Instantiating RAMAC with a diffusion actor, we illustrate these insights on a 2-D risky bandit and evaluate on Stochastic-D4RL, observing consistent gains in $\mathrm{CVaR}_{0.1}$ while maintaining strong returns. The code and experimental results are available on the \href{https://kaifukazawa.github.io/ramac-project/} {project website}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
