QHyer: 用于离线目标条件强化学习的Q-条件混合注意力-门控Transformer
QHyer: Q-conditioned Hybrid Attention-mamba Transformer for Offline Goal-conditioned RL
AI总结：
本文提出QHyer，通过引入状态条件化的Q估计器和混合注意力-门控架构，解决离线目标条件强化学习中长期依赖建模和稀疏奖励下的行为拼接问题，验证了其在非马尔可夫和马尔可夫数据集上的优越性能。
摘要：
离线目标条件强化学习（GCRL）从静态数据集中学习目标到达策略，但现实数据集常为部分可观测且历史依赖，呈现马尔可夫与非马尔可夫混合特性，违反标准强化学习假设。历史感知序列模型如决策变压器（DT）适合长期依赖建模，但纯注意力在处理局部马尔可夫结构和长程上下文时效率低下且脆弱。尽管最近的混合架构（如LSDT）引入局部提取器改进局部依赖建模，但固定窗口提取无法适应不同时间异质设置中变化的依赖长度，常截断长程上下文而非自适应压缩内容。此外，序列离线GCRL面临关键瓶颈：在稀疏奖励下，返回到目标（RTG）在子轨迹间变得非判别性，提供很少的指导信号以拼接不同演示中的目标到达行为。为解决这些问题，我们提出QHyer，用流参数化的状态条件化目标到达Q估计器替代RTG以支持跨演示拼接，并引入门控混合注意力-门控架构，实现内容自适应的历史压缩同时保留局部动态。广泛实验表明，QHyer在非马尔可夫和马尔可夫数据集上均取得最先进性能，验证了其在多样化场景中的有效性。
英文摘要：
Offline goal-conditioned RL (GCRL) learns goal-reaching policies from static datasets, but real-world datasets are often partially observable and history-dependent, exhibiting a mix of Markovian and non-Markovian that violate standard RL assumptions. History-aware sequence models such as Decision Transformer (DT) are a natural fit for long-term dependency modeling, yet pure attention is inefficient and brittle when handling local Markovian structure and long-range context simultaneously. Although recent hybrid architectures (e.g., LSDT) introduce local extractors to improve local dependencies modeling, the fixed-window extraction cannot adapt its effective memory to varying dependency lengths in temporally heterogeneous settings, often truncating long-range context rather than compressing its content adaptively. Moreover, sequential offline GCRL faces a key bottleneck: under sparse rewards, return-to-go (RTG) becomes non-discriminative across sub-trajectories, providing little guidance signal for stitching goal-reaching behaviors from diverse demonstrations. To address these, we propose \textbf{QHyer}, which replaces RTG with a flow-parameterized, state-conditioned goal-reaching Q-estimator to support stitching across demonstrations, and introduces a gated Hybrid Attention-Mamba backbone that performs content-adaptive history compression while preserving local dynamics. Extensive experiments demonstrate that \textbf{QHyer} achieves state-of-the-art performance on both non-Markovian and Markovian datasets, validating its effectiveness for diverse scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
