迹介导的峰值偏差：深度强化学习中时间信用分配与认知启发式的桥梁
Trace-Mediated Peak Bias: Bridging Temporal Credit Assignment and Cognitive Heuristics in Deep Reinforcement Learning
AI总结：
本文发现深度强化学习中的迹介导峰值偏差（TMPB），揭示了其作为峰值-末端规则的机制基础，并证明自适应优化器通过二阶矩归一化可缓解该偏差。
摘要：
时间信用分配是生物和人工智能的核心问题，但其与非线性函数逼近的相互作用尚不清楚。我们在深度强化学习中识别出一种系统性失效模式，称为迹介导峰值偏差（TMPB）。在中间资格迹深度下，智能体非理性地偏好具有高幅度奖励“峰值”的轨迹，而非具有更高累积回报的替代轨迹。这为峰值-末端规则提供了一种机制解释：一种人类记忆偏差，其中经验由其最强烈的时刻而非整合效用判断。我们证明，TMPB的出现是因为迹将远时时间差分误差放大为“梯度冲击”，而固定步长的随机梯度下降无法将其归一化，导致全局高估。相反，自适应优化器通过二阶矩归一化缓解了这种病理现象。我们的结果表明，类人的显著性扭曲可能自然产生于分布式系统中信用分配的数学约束，而自适应优化是理性价值估计的理论必要条件。
英文摘要：
Temporal credit assignment is central to both biological and artificial intelligence, yet its interaction with non-linear function approximation is poorly understood. We identify a systematic failure mode in deep reinforcement learning (RL) termed Trace-Mediated Peak Bias (TMPB). At intermediate eligibility trace depths, agents irrationally prefer trajectories with high-magnitude reward ``peaks'' over alternatives with higher cumulative returns. This provides a mechanistic account of the Peak-End Rule: a human memory bias where experiences are judged by their most intense moments rather than integrated utility. We show that TMPB emerges because traces amplify distal Temporal Difference errors into ``gradient shocks'' that fixed-step-size Stochastic Gradient Descent cannot normalize, leading to global overestimation. Conversely, adaptive optimizers mitigate this pathology via second-moment normalization. Our results suggest that human-like saliency distortions may emerge naturally from the mathematical constraints of credit assignment in distributed systems, and that adaptive optimization is a theoretical necessity for rational value estimation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
