迈向可扩展的多任务强化学习与大决策模型
Towards Scalable Multi-Task Reinforcement Learning with Large Decision Models
AI总结：
提出LDM-v0，一个在大规模异构强化学习环境上离线预训练的Transformer策略，通过监督下一动作预测实现多任务学习，在约1000个环境中达到与独立训练策略相当的性能。
摘要：
近期大规模序列建模的进展表明，单个模型可以在高度多样化的数据分布中学习有用的表示。受这些进展启发，我们研究是否可以在大量异构强化学习环境的集合上训练统一的Transformer策略。我们引入LDM-v0，一个在来自数千个跨多个领域和模态的环境的轨迹上离线训练的大决策模型。LDM-v0是一个多任务、多模态的Transformer策略，以观测、动作、奖励和终止信号的历史为条件，并通过离线轨迹上的监督下一动作预测进行训练。我们描述了用于构建LDM-v0的环境基础设施、自动化数据生成流程、模型架构和训练方法，并评估了其在多样环境中的性能。我们表明，单个预训练模型在约1000个环境中（包括机器人、自动驾驶、库存管理、网络安全、交易和视频游戏）与独立训练的任务特定参考策略性能相当。这些结果证明了使用单个Transformer策略在异构强化学习环境中进行大规模离线预训练的可行性。
英文摘要：
Recent progress in large-scale sequence modeling has shown that a single model can learn useful representations across highly diverse data distributions. Inspired by these advances, we investigate whether a unified transformer policy can be trained across large collections of heterogeneous reinforcement learning environments. We introduce LDM-v0, a Large Decision Model trained offline on trajectories collected from thousands of environments spanning multiple domains and modalities. LDM-v0 is a multi-task, multi-modal transformer policy conditioned on histories of observations, actions, rewards, and termination signals, and trained through supervised next-action prediction over offline trajectories. We describe the environment infrastructure, automated data generation pipeline, model architecture, and training methodology used to build LDM-v0, and evaluate its performance across diverse environments. We show that a single pretrained model matches the performance of independently trained task-specific reference policies on approximately 1,000 environments including robotics, autonomous driving, inventory management, cybersecurity, trading, and video games. These results demonstrate the feasibility of large-scale offline pretraining across heterogeneous reinforcement learning environments using a single transformer policy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
