arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.08892cs.AI
收藏

Lumine:构建3D开放世界通才智能体的开放方案

Lumine: An Open Recipe for Building Generalist Agents in 3D Open Worlds

ByteDance Seed(字节跳动种子);NTU(国立科技大学)

Weihao Tan, Xiangyang Li, Yunhao Fang, Heyuan Yao, Shi Yan, Hao Luo, Tenglong Ao, Huihui Li, Hongbin Ren, Bairen Yi, Yujia Qin, Bo An, Libin Liu, Guang Shi

AI总结:

该研究提出首个开放方案Lumine,通过端到端类人交互范式统一感知、推理和行动,在《原神》中完成五小时蒙德主线且效率达人类水平,还能零样本泛化至《鸣潮》和《崩坏:星穹铁道》,推动开放式环境通才智能体发展。

摘要:

我们介绍Lumine,这是首个用于开发通才智能体的开放方案,该智能体能够在具有挑战性的3D开放世界环境中实时完成长达数小时的复杂任务。Lumine采用类人交互范式,以端到端方式统一感知、推理和行动,由视觉-语言模型提供支持。它以5 Hz的频率处理原始像素,生成精确的30 Hz键盘-鼠标动作,并仅在必要时自适应调用推理。在《原神》中训练后,Lumine成功完成了整个五小时的蒙德主线剧情,效率与人类水平相当,并能遵循自然语言指令,在3D开放世界探索和2D GUI操作中执行广泛任务,涵盖收集、战斗、解谜和NPC交互。除域内性能外,Lumine还展现出强大的零样本跨游戏泛化能力。无需任何微调,它就能在《鸣潮》中完成100分钟的任务,并在《崩坏:星穹铁道》中完成整个五小时的第一章内容。这些令人鼓舞的结果突显了Lumine在不同世界和交互动态中的有效性,标志着向开放式环境中通才智能体迈出了坚实一步。

英文摘要:

We introduce Lumine, the first open recipe for developing generalist agents capable of completing hours-long complex missions in real time within challenging 3D open-world environments. Lumine adopts a human-like interaction paradigm that unifies perception, reasoning, and action in an end-to-end manner, powered by a vision-language model. It processes raw pixels at 5 Hz to produce precise 30 Hz keyboard-mouse actions and adaptively invokes reasoning only when necessary. Trained in Genshin Impact, Lumine successfully completes the entire five-hour Mondstadt main storyline on par with human-level efficiency and follows natural language instructions to perform a broad spectrum of tasks in both 3D open-world exploration and 2D GUI manipulation across collection, combat, puzzle-solving, and NPC interaction. In addition to its in-domain performance, Lumine demonstrates strong zero-shot cross-game generalization. Without any fine-tuning, it accomplishes 100-minute missions in Wuthering Waves and the full five-hour first chapter of Honkai: Star Rail. These promising results highlight Lumine's effectiveness across distinct worlds and interaction dynamics, marking a concrete step toward generalist agents in open-ended environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑