arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.04513cs.AI
收藏

BiTAgent: 一种面向任务的模块化框架,用于多模态大语言模型与世界模型之间的双向耦合

BiTAgent: A Task-Aware Modular Framework for Bidirectional Coupling between Multimodal Large Language Models and World Models

Tsinghua University(清华大学);Shandong University(山东大学)

Yu-Wei Zhan, Xin Wang, Pengzhe Mao, Tongtong Feng, Ren Wang, Wenwu Zhu

AI总结:

BiTAgent通过双向耦合多模态大语言模型与世界模型,实现任务感知的动态联合学习,提升多任务和跨环境的泛化能力。

摘要:

构建通用的具身代理需要一个统一的系统,能够解释多模态目标,建模环境动态,并在多样化的现实任务中执行可靠的行动。多模态大语言模型(MLLMs)提供了强大的语义先验和跨模态泛化能力,而世界模型(WMs)提供了可操作的潜在动态用于预测和控制。它们的结合对于开放式的具身智能具有前景,但引入了两个关键挑战:(1)在WM的潜在空间中建立MLLM的语义意图与动态状态表示之间的紧密耦合,以及(2)实现任务感知的适应性,以支持多任务学习和跨环境泛化。为了解决这些限制,我们提出了BiTAgent,一种面向任务的动态联合框架,能够实现MLLMs和WMs之间的双向耦合。BiTAgent建立了两条互补的路径:一条前向路径将MLLM的表示注入到WM的潜在空间中,以实现语义引导的想象;另一条后向路径中,WM生成的反馈通过密集的文本条件奖励来优化MLLM的语义空间。这种双向交互通过三个协同的组件实现:面向任务的动态联合学习、面向任务的行为学习以及MLLM-WM联合优化,这些共同协调语义推理和动态预测。在多任务和跨环境设置中的广泛实验表明,BiTAgent在稳定性和泛化能力上优于最先进的基线,标志着开放式的具身学习迈出了一步。

英文摘要:

Building generalist embodied agents requires a unified system that can interpret multimodal goals, model environment dynamics, and execute reliable actions across diverse real-world tasks. Multimodal large language models (MLLMs) offer strong semantic priors and cross-modal generalization, while world models (WMs) provide actionable latent dynamics for prediction and control. Their combination holds promise for open-ended embodied intelligence, yet introduces two key challenges: (1) establishing a tight coupling between the semantic intent from MLLMs and the dynamic state representations within the WM's latent space, and (2) achieving task-aware adaptability that supports multi-task learning and cross-environment generalization. To address these limitations, we propose BiTAgent, a task-aware dynamic joint framework that enables bidirectional coupling between MLLMs and WMs. BiTAgent establishes two complementary pathways: a forward path that injects MLLM representations into the WM's latent space for semantically guided imagination, and a backward path where WM-generated feedback refines the MLLM's semantic space via dense text-conditioned rewards. This bidirectional interaction is realized through three synergistic components: Task-Aware Dynamic Joint Learning, Task-Aware Behavior Learning, and MLLM-WM Joint Optimization, which together harmonize semantic reasoning and dynamic prediction. Extensive experiments across multi-task and cross-environment settings demonstrate superior stability and generalization over state-of-the-art baselines, marking a step toward open-ended embodied learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑