自确认Transformer：离线多智能体强化学习中的信念条件化自适应
Self-Confirming Transformer for Belief-Conditioned Adaptation in Offline Multi-Agent Reinforcement Learning
AI总结：
针对离线多智能体强化学习中非平稳对手导致的分布偏移问题，提出自确认Transformer（SCT），通过自增强预条件训练实现信念条件化在线自适应，在迭代囚徒困境和多粒子环境中验证了优越性能。
摘要：
离线强化学习（RL）面临离线数据集与在线环境之间的分布偏移问题。在多智能体强化学习（MARL）中，这种分布偏移可能源于在线测试中非平稳对手的行为与离线数据集中记录的行为不同。因此，离线MARL更广泛应用的关键在于对非平稳对手的在线自适应。近期基础模型（如大语言模型）的进展展示了Transformer（一种新兴的神经网络架构）在序列建模中的泛化能力，而离线RL是序列建模的一个特例。人们自然好奇：基于离线训练的Transformer RL策略能否在线适应非平稳对手？我们提出了一种新颖的自回归训练方法，基于自增强预条件的思想，赋予Transformer智能体在线自适应能力。Transformer智能体首先离线学习基于过去观测预测对手动作。在线部署时，这种虚构的对手行为（称为信念）与其他环境反馈一起被反馈给Transformer，以生成条件于该信念的未来动作。受博弈论中自确认均衡的启发，训练损失包括信念一致性损失（要求信念与对手实际动作匹配）和最优反应损失（要求智能体在信念下表现最优）。我们在结构化环境——迭代囚徒困境博弈中评估了所提出的自确认Transformer（SCT）的在线自适应能力，以展示SCT的信念一致性和均衡行为，并在更复杂的多粒子环境中展示其相对于先前Transformer和离线MARL基线在应对非平稳对手时的优越性能。
英文摘要：
Offline reinforcement learning (RL) suffers from the distribution shift between the offline dataset and the online environment. In multi-agent RL (MARL), this distribution shift may arise from the nonstationary opponents in the online testing who display distinct behaviors from those recorded in the offline dataset. Hence, the key to the broader deployment of offline MARL is the online adaptation to nonstationary opponents. Recent advances in foundation models, e.g., large language models, have demonstrated the generalization ability of the transformer, an emerging neural network architecture, in sequence modeling, of which offline RL is a special case. One naturally wonders \textit{whether offline-trained transformer-based RL policies adapt to nonstationary opponents online}. We propose a novel auto-regressive training to equip transformer agents with online adaptability based on the idea of self-augmented pre-conditioning. The transformer agent first learns offline to predict the opponent's action based on past observations. When deployed online, such a fictitious opponent play, referred to as the belief, is fed back to the transformer, together with other environmental feedback, to generate future actions conditional on the belief. Motivated by self-confirming equilibrium in game theory, the training loss consists of belief consistency loss, requiring the beliefs to match the opponent's actual actions and best response loss, mandating the agent to behave optimally under the belief. We evaluate the online adaptability of the proposed self-confirming transformer (SCT) in a structured environment, iterated prisoner's dilemma games, to demonstrate SCT's belief consistency and equilibrium behaviors as well as more involved multi-particle environments to showcase its superior performance against nonstationary opponents over prior transformers and offline MARL baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
