MemAct：结合记忆与主动探索的流空间理解
MemAct: Streaming Spatial Understanding with Memory and Active Exploration
AI总结：
MemAct是结合记忆与主动探索的流空间理解框架，通过视觉-语言模型实现记忆读写与主动探索，在MemAct-bench上优于现有基线，兼具高效性与前沿性能。
摘要：
空间理解对于具身智能体在物理世界中导航和交互至关重要，由于观测是逐步到达的，智能体必须以流的方式进行推理。然而，流理解也必须是闭环的：与任务相关的证据既需要回忆过去的观测，也需要主动获取新的视角。现有的流方法大多侧重于处理传入的观测，而任务驱动的主动探索却未得到充分研究。我们提出MemAct，一个结合记忆与动作的流空间理解框架。共享的视觉-语言模型会逐步写入结构化的历史记忆，按需检索记录和过去的帧，并通过移动和转向获取额外证据。检索到的证据和动作观测会为后续决策提供信息，使智能体能够在统一的推理循环中继续探索或给出答案。我们开发了MemAct套件，这是一个跨真实和模拟环境的数据生成流水线与集合，包含约100万个用于空间推理、记忆写入、检索和探索的训练样本。其MemAct-bench包含2万组主要问答对，用于评估当前观测理解、历史证据使用和主动探索，以及2000个独立的时序问题。MemAct结合了针对记忆与交互的监督微调，以及利用累积证据增益、动作成本和最终答案质量的反事实动作优化，以改进探索和停止决策。实验表明，MemAct在MemAct-bench上的表现优于现有的空间理解和流基线方法，且接近前沿多模态模型，同时保持了流推理效率。
英文摘要：
Spatial understanding is essential for embodied agents navigating and interacting with the physical world. As observations arrive incrementally, agents must reason in a streaming manner. Yet streaming understanding must also be closed-loop: task-relevant evidence may require both recalling past observations and actively acquiring new views. Existing streaming approaches largely emphasize processing incoming observations, leaving task-driven active exploration underexplored. We introduce MemAct, a framework for streaming spatial understanding with memory and action. A shared vision--language model incrementally writes structured historical memory, retrieves records and past frames on demand, and acquires additional evidence through movement and turning. Retrieved evidence and action observations inform subsequent decisions, enabling the agent to continue exploring or answer within a unified reasoning loop. We develop the MemAct suite, a data generation pipeline and collection across real and simulated environments, with approximately 1M training examples for spatial reasoning, memory writing, retrieval, and exploration. Its MemAct-bench comprises 20,000 main QA pairs evaluating current-observation understanding, historical evidence use, and active exploration, plus 2,000 separate Timing questions. MemAct combines supervised fine-tuning for memory and interaction with counterfactual action optimization using cumulative evidence gains, action costs, and terminal answer quality to improve exploration and stopping decisions. Experiments show that MemAct outperforms existing spatial understanding and streaming baselines and approaches frontier multimodal models on MemAct-bench while retaining streaming inference efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
