arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节(Veterans Day),12 日无更新
arXiv 2610.22686cs.CL
收藏

面向长视界具身智能体的多模态文件系统记忆

Multimodal Filesystem Memory For Long-Horizon Embodied Agents

Harvard(哈佛大学);Kempner Institute(坎普纳研究所);NYU(纽约大学);UIUC(伊利诺伊大学厄巴纳-香槟分校);NEU(东北大学);MIT(麻省理工学院)

Chloe Su, Jiahan Li, Zhenting Qi, Chaoran Cheng, Haoyu Chen, Sejoon Jun, Weirui Ye, Yilun Du

AI总结:

本文提出多模态文件系统记忆(MFM)架构,通过整合多类型记忆并结合文本检索,提升了长视界具身智能体在Minecraft、CUABench-KiCad等任务中的性能。

摘要:

在长视界交互中运行的智能体需要有效记忆,以在整个交互过程中复用经验。若保留全部交互历史,会难以找到相关经验;若将其压缩为摘要,则可能丢失后续所需的视觉与空间细节。本文提出多模态文件系统记忆架构(MFM),该架构保留互补类型的智能体记忆:情景记忆存储观测、动作与结果;空间记忆将经验与位置关联;语义与程序记忆构建通用知识与可复用行动方法;工作记忆存储当前计划。MFM利用可检索的文本记录,将这些不同表征与底层多模态情景关联。智能体可通过上下文线索查找过往经验,并检查原始观测以恢复文本描述中缺失的细节。通过整合,智能体将经验转化为更高层次的知识与程序,同时仍保留对原始情景的引用。在长视界Minecraft及计算机使用任务中,MFM使三个基础模型的性能提升:使用GPT-6-Astra时,MFM完成“从零开始开采钻石”任务,在“末影龙”任务中达成24个里程碑中的16个,而最佳基线为12.7;在CUABench-KiCad上,其平均奖励为0.440,最佳基线为0.400,无持久记忆时为0.267。

英文摘要:

Agents operating over long horizons need an effective memory that allows them to reuse their experience throughout long interactions. If the entire history of an interaction is retained, it becomes difficult to find the relevant experience, while compressing it into summaries may result in the loss of visual and spatial details that are required later on. We present \textbf{MFM}, a multimodal filesystem memory architecture that keeps complementary types of agent memory. \textbf{Episodic memory} stores the observations, actions, and outcomes, and \textbf{spatial memory} associates experiences with locations. \textbf{Semantic and procedural memory} build up general knowledge and reusable methods of acting, and \textbf{working memory} holds the current plans. MFM makes use of searchable textual records to connect these different representations to the underlying multimodal episodes. The agents are able to find previous experiences by means of contextual cues and to examine the original observations in order to recover details that are missing from their textual descriptions. Through consolidation, the agents convert experience into higher-level knowledge and procedures while still keeping references to the original episodes. In the case of long-horizon Minecraft and computer-use tasks, MFM leads to improved performance with three base models. With GPT-6-Astra, MFM completes \emph{Mine Diamond from Scratch} and reaches 16 out of 24 milestones on the \emph{Ender Dragon} task, as against 12.7 for the best baseline. On CUABench-KiCad it attains a mean reward of 0.440, as compared to 0.400 for the best baseline and 0.267 in the case of no persistent memory.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑