MLLM作为检索器：交互式学习具身智能体的多模态检索
MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agents
AI总结：
针对现有检索忽视轨迹有效性，提出MART方法，利用交互数据微调MLLM检索器，结合轨迹抽象，显著提升具身智能体在未见场景的任务成功率。
摘要：
MLLM智能体通过检索多模态任务相关轨迹数据，展现出处理复杂具身任务的潜力。然而，当前检索方法主要关注轨迹中文本或视觉线索的表面相似性，忽略了它们对当前特定任务的有效性。为解决这一问题，我们提出了一种新方法，MLLM作为检索器（MART），该方法利用交互数据，基于偏好学习微调MLLM检索器，从而增强具身智能体的性能，使检索器充分考虑轨迹的有效性，并优先为未见任务选择轨迹。我们还引入了轨迹抽象机制，利用MLLM的总结能力，用更少的标记表示轨迹，同时保留关键信息，使智能体更好地理解轨迹中的里程碑。在多种环境下的实验结果表明，与基线方法相比，我们的方法显著提高了未见场景中的任务成功率。这项工作通过将通用MLLM微调为检索器来评估轨迹有效性，为具身智能体的多模态检索提供了新范式。所有基准任务代码、模拟器修改和MLLM检索器均可在https://github.com/PKU-RL/MART获取。
英文摘要：
MLLM agents demonstrate potential for complex embodied tasks by retrieving multimodal task-relevant trajectory data. However, current retrieval methods primarily focus on surface-level similarities of textual or visual cues in trajectories, neglecting their effectiveness for the specific task at hand. To address this issue, we propose a novel method, MLLM As ReTriever (MART), which enhances the performance of embodied agents by utilizing interaction data to fine-tune an MLLM retriever based on preference learning, such that the retriever fully considers the effectiveness of trajectories and prioritizes them for unseen tasks. We also introduce Trajectory Abstraction, a mechanism that leverages MLLMs' summarization capabilities to represent trajectories with fewer tokens while preserving key information, enabling agents to better comprehend milestones in the trajectory. Experimental results across various environments demonstrate our method significantly improves task success rates in unseen scenes compared to baseline methods. This work presents a new paradigm for multimodal retrieval in embodied agents, by fine-tuning a general-purpose MLLM as the retriever to assess trajectory effectiveness. All the code for benchmark tasks, simulator modifications, and the MLLM retriever is available at https://github.com/PKU-RL/MART.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
