arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.02664cs.AIcs.MA
收藏

通过生成式世界模型为多智能体决策问题提供有依据的答案

Grounded Answers for Multi-agent Decision-making Problem through Generative World Model

National Engineering Research Center for Visual Information and Application(国家视觉信息与应用工程研究中心);Institute of Artificial Intelligence and Robotics(人工智能与机器人研究所);Xi’an Jiaotong University(西安交通大学)

Zeyang Liu, Xinrui Yang, Shiguang Sun, Long Qian, Lipeng Wan, Xingyu Chen, Xuguang Lan

AI总结:

本文提出将语言引导的世界模型模拟器集成到多智能体强化学习中,通过分别学习动态和奖励模型来生成有依据的决策答案,并在SMAC基准上验证了其优越性。

摘要:

生成模型的最新进展在图像生成和聊天机器人等许多领域激发了重大创新。尽管取得了成功,这些模型在复杂的多智能体决策问题上往往会产生粗略且具有误导性的解决方案,因为它们缺乏人类那样的试错经验和推理能力。为了解决这一局限,我们探索了一种将语言引导的模拟器集成到多智能体强化学习流程中的范式,以增强生成的答案。该模拟器是一个分别学习动态和奖励的世界模型,其中动态模型包含一个图像分词器和一个因果Transformer,用于自回归地生成交互转移,而奖励模型是一个双向Transformer,通过在语言引导下最大化专家演示中轨迹的似然来学习。给定当前状态的图像和任务描述,我们使用世界模型训练联合策略,并通过在动态模型上运行收敛后的策略来生成图像序列作为答案。实证结果表明,该框架通过在StarCraft Multi-Agent Challenge基准的训练任务和未见任务上展现出优越性能,能够改善多智能体决策问题的答案。特别是,它可以生成一致的交互序列,并在交互状态下生成可解释的奖励函数,为训练未来的生成式模型开辟了道路。

英文摘要:

Recent progress in generative models has stimulated significant innovations in many fields, such as image generation and chatbots. Despite their success, these models often produce sketchy and misleading solutions for complex multi-agent decision-making problems because they miss the trial-and-error experience and reasoning as humans. To address this limitation, we explore a paradigm that integrates a language-guided simulator into the multi-agent reinforcement learning pipeline to enhance the generated answer. The simulator is a world model that separately learns dynamics and reward, where the dynamics model comprises an image tokenizer as well as a causal transformer to generate interaction transitions autoregressively, and the reward model is a bidirectional transformer learned by maximizing the likelihood of trajectories in the expert demonstrations under language guidance. Given an image of the current state and the task description, we use the world model to train the joint policy and produce the image sequence as the answer by running the converged policy on the dynamics model. The empirical results demonstrate that this framework can improve the answers for multi-agent decision-making problems by showing superior performance on the training and unseen tasks of the StarCraft Multi-Agent Challenge benchmark. In particular, it can generate consistent interaction sequences and explainable reward functions at interaction states, opening the path for training generative models of the future.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑