发表机构
Information Sciences Institute; University of Southern California; Capital One（信息科学研究所; 南加州大学; 第一资本）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究系统比较了微调与RAG在基于语言模型的世界建模中的性能，发现微调通常更优，但RAG更数据高效，并提出混合系统结合两者优势，在多个环境中表现最佳。
AI 中文摘要
世界模型（WMs）模拟环境的转移动态，使智能体能够根据其行为的后果进行规划。在基于文本的环境中，微调语言模型（LM）以充当世界模型已成为一种主导范式。然而，尽管诸如检索增强生成（RAG）之类的非参数方法取得了广泛成功，但基于检索的LM世界建模仍未得到充分探索。我们在涵盖具身、网页导航和社交设置的五个不同环境中进行了系统评估，比较了基于微调和基于RAG的LM世界建模方法。我们的研究表明，微调通常优于RAG，微调后的世界模型使智能体在15/20个设置中获得更高奖励。虽然两种构建范式都受益于更多样化的探索，但基于RAG的方法被证明更具数据效率，而微调方法则不成比例地受益于扩大收集经验的数量。我们专注于基于RAG的世界模型，设计了一种使用反事实干预来估计检索阶段错误率的程序，并表明检索器始终从经验缓冲区中浮现次优转移。为了解决这一缺陷，我们研究了各种查询重构策略，证明了一种分层方法优于传统检索流程。最后，我们将我们的发现整合到一个混合世界建模系统中，该系统参数化地捕获核心环境动态，同时学习依赖从主动维护的记忆存储中检索。我们的混合系统在多个环境和模型中始终优于其他方法，展示了该方法的稳健性和我们发现的适用性。
英文摘要
World models (WMs) simulate the transition dynamics of environments, enabling agents to plan over the consequences of their actions. In text-based environments, fine-tuning a Language Model (LM) to serve as a WM has emerged as a dominant paradigm. However, despite the widespread success of non-parametric approaches such as Retrieval Augmented Generation (RAG), retrieval for LM-based world modelling remains underexplored. We conduct a systematic evaluation across five diverse environments spanning embodied, web navigation and social settings, comparing fine-tuning and RAG-based approaches for LM-based world modelling. Our study reveals that fine-tuning often outperforms RAG, with fine-tuned WMs enabling agents to obtain higher rewards on 15/20 settings. While both construction paradigms benefit from additional and more diverse exploration, RAG-based approaches prove more data-efficient, and fine-tuning approaches disproportionately benefit from scaling the amount of experience collected. With a focus on RAG-based WMs, we devise a procedure that uses counterfactual intervention to estimate the error rate of the retrieval stage, and show that retrievers consistently surface suboptimal transitions from the experience buffer. Hoping to address this failing, we study a variety of query reformulation strategies, demonstrating that a hierarchical approach outperforms the traditional retrieval pipeline. Finally, we compose our findings into a hybrid world modelling system that parametrically captures core environment dynamics, while learning to rely on retrieval from an actively maintained memory store. Our hybrid system consistently outperforms other methods across multiple environments and models, showcasing the robustness of the approach and the applicability of our findings.
