UNIQ: 离线强化学习中的自适应保守性共形校准
UNIQ: Conformal Calibration for Adaptive Conservatism in Offline Reinforcement Learning
AI总结：
提出UNIQ方法，通过共形预测校准不确定性，实现状态自适应的保守性惩罚，在D4RL基准上以接近IQL的内存开销提升性能。
摘要：
离线强化学习需要谨慎的保守性来缓解分布偏移，然而大多数现有方法在所有状态上统一施加固定惩罚，而不考虑局部数据覆盖。我们提出UNIQ（不确定性信息分位数），一种通过共形校准不确定性估计引入状态自适应保守性的离线RL方法。基于隐式Q学习（IQL）主干，UNIQ训练一个多期望值集成，使用分裂共形预测计算无分布不确定性估计，并将所得信号映射到状态依赖的期望值，从而在覆盖良好的区域放松保守性，在数据边界附近的不确定区域加强保守性。在D4RL MuJoCo基准上，UNIQ持续优于IQL，在Walker2d和重放密集型任务上提升最大。同时，UNIQ以接近IQL的内存成本（约250 MB峰值VRAM）运行，相比EDAC提供约10倍的减少。我们不追求整体最先进性能，而是将UNIQ定位为一种实用机制贡献，改进了离线强化学习中的性能-效率权衡。
英文摘要：
Offline reinforcement learning requires careful conservatism to mitigate distribution shift, yet most existing methods apply a fixed penalty uniformly across all states regardless of local data coverage. We present UNIQ (Uncertainty-Informed Quantile), an offline RL method that introduces state-adaptive conservatism through conformally calibrated uncertainty estimation. Built on the Implicit Q-Learning (IQL) backbone, UNIQ trains a multi-expectile value ensemble, computes distribution-free uncertainty estimates using split conformal prediction, and maps the resulting signal to a state-dependent expectile that relaxes conservatism in well-covered regions while strengthening it in uncertain regions near the data frontier. On D4RL MuJoCo benchmarks, UNIQ consistently improves over IQL, with the largest gains observed on Walker2d and replay-heavy tasks. At the same time, UNIQ operates at near-IQL memory cost (approximately 250 MB peak VRAM), providing roughly a 10x reduction compared to EDAC. Rather than pursuing overall state-of-the-art performance, we position UNIQ as a practical mechanism contribution that improves the performance-efficiency trade-off in offline reinforcement learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
