arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.31075cs.CV
收藏

面向多模态智能体的任务聚焦记忆

Task-Focused Memorization for Multimodal Agents

Fudan University(复旦大学)

Tao Zou, Yichen He, Tian Qiu, Yuan Lin, Hang Li

AI总结:

提出基于强化学习的任务聚焦记忆策略学习框架TaskMem,通过两阶段训练使多模态智能体在流式观测中动态选择任务相关记忆,在三个流式基准上VQA准确率提升5.3%-7.0%。

摘要:

长期记忆对于多模态智能体构建连贯经验、积累世界知识和实现持续学习至关重要。然而,构建有效记忆不仅涉及记忆模块设计和准确性、保真度等基本要求,关键挑战在于决定记忆什么。多模态智能体(如具身智能体)在真实或虚拟环境中持续感知、推理和行动,接收无界的多模态观测流。面对这种信息组合爆炸,智能体必须选择性地保留与其环境角色相关且对未来任务有价值的内容。为弥合这一差距,我们将记忆生成建模为可学习的记忆策略,并引入TaskMem(任务聚焦记忆策略学习),一种基于强化学习的框架,使策略能够动态调整其关注点以适应环境中遇到的实际任务需求。TaskMem采用两阶段训练范式:第一阶段在基本保真度要求下优化记忆质量,学习如何记忆;第二阶段在部署后进行,智能体通过在其基础MLLM上调整适配器来学习记忆什么,利用近期环境任务定义奖励模型,引导记忆策略聚焦于任务相关的内容。为评估我们的方法,我们将VideoMME、EgoLife和EgoTempo重新构建为流式基准,模拟智能体处理流式观测并处理在线到达任务的真实场景。为隔离记忆评估,问题必须仅使用智能体的记忆回答,而不访问原始视频。基于Qwen3-VL-30B-A3B,TaskMem在这些基准上分别将VQA准确率提高了6.3%、7.0%和5.3%。

英文摘要:

Long-term memory is essential for multimodal agents to build coherent experience, accumulate world knowledge, and achieve continual learning. However, constructing effective memory goes beyond memory module design and basic requirements such as accuracy and fidelity; the key challenge lies in determining what to memorize. Multimodal agents, such as embodied agents, continuously perceive, reason, and act in real or virtual environments, receiving an unbounded stream of multimodal observations. From this combinatorial explosion of information, an agent must selectively retain content that is relevant to its role in the environment and valuable for future tasks. To bridge this gap, we frame memory generation as a learnable memorization policy and introduce TaskMem (Task-focused Memorization Policy Learning), a reinforcement-learning-based framework that enables the policy to dynamically adjust its focus to the demands of real tasks encountered in the environment. TaskMem adopts a two-phase training paradigm: Phase One learns how to memorize by optimizing memory quality under fundamental fidelity requirements; Phase Two occurs after deployment, where the agent learns what to memorize by tuning an adapter on its base MLLM, using recent environment tasks to define a reward model that guides the memorization policy toward task-relevant content. To evaluate our approach, we reformulate VideoMME, EgoLife, and EgoTempo into streaming benchmarks that simulate a realistic setting in which an agent processes streaming observations and handles tasks arriving online. To isolate memory assessment, the questions must be answered using only the agent's memory, without access to raw video. Built on Qwen3-VL-30B-A3B, TaskMem improves VQA accuracy by 6.3%, 7.0%, and 5.3% on these benchmarks, respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑