AIC MLLM：用于鲁棒机器人操作的自主交互纠错MLLM
AIC MLLM: Autonomous Interactive Correction MLLM for Robust Robotic Manipulation
AI总结：
提出AIC MLLM框架，通过微调获取位姿预测与反馈提示能力，利用视觉掩码和文本描述纠正铰接物体操作中的低级SE(3)接触位姿，结合反馈信息提取与测试时适应策略，有效提升了机器人在模拟与真实环境中的操作鲁棒性。
摘要：
反思和纠正失败的能力对于机器人系统与现实生活中的物体进行稳定交互至关重要。观察到多模态大语言模型（MLLMs）的泛化和推理能力，以往的方法旨在利用这些模型来相应地增强机器人系统。然而，这些方法通常侧重于使用额外的MLLM进行高级规划纠错，对失败样本的利用有限，无法纠正低级接触位姿，而这在铰接物体操作中尤其容易发生。为了填补这一空白，我们提出了自主交互纠错（AIC）MLLM，它利用先前的低级交互经验来纠正铰接物体的SE(3)位姿预测。具体而言，AIC MLLM最初被微调以获取位姿预测和反馈提示理解能力。我们设计了两种用于与物体交互的提示指令：1）视觉掩码以突出不可移动的部分进行位置纠正，2）文本描述以指示旋转纠正的潜在方向。在推理过程中，引入了反馈信息提取模块来识别失败原因，使AIC MLLM能够使用相应的提示自适应地纠正位姿预测。为了进一步增强操作稳定性，我们设计了一种测试时适应策略，使AIC MLLM能够更好地适应当前的场景配置。最后，在模拟和真实世界环境中进行了大量实验以评估所提出的方法。结果表明，我们的AIC MLLM可以通过利用交互经验提示有效地纠正失败样本。我们的项目网站是https://sites.google.com/view/aic-mllm。
英文摘要：
The ability to reflect on and correct failures is crucial for robotic systems to interact stably with real-life objects.Observing the generalization and reasoning capabilities of Multimodal Large Language Models (MLLMs), previous approaches have aimed to utilize these models to enhance robotic systems accordingly.However, these methods typically focus on high-level planning corrections using an additional MLLM, with limited utilization of failed samples to correct low-level contact poses which is particularly prone to occur during articulated object manipulation.To address this gap, we propose an Autonomous Interactive Correction (AIC) MLLM, which makes use of previous low-level interaction experiences to correct SE(3) pose predictions for articulated object. Specifically, AIC MLLM is initially fine-tuned to acquire both pose prediction and feedback prompt comprehension abilities.We design two types of prompt instructions for interactions with objects: 1) visual masks to highlight unmovable parts for position correction, and 2) textual descriptions to indicate potential directions for rotation correction. During inference, a Feedback Information Extraction module is introduced to recognize the failure cause, allowing AIC MLLM to adaptively correct the pose prediction using the corresponding prompts.To further enhance manipulation stability, we devise a Test Time Adaptation strategy that enables AIC MLLM to better adapt to the current scene configuration.Finally, extensive experiments are conducted in both simulated and real-world environments to evaluate the proposed method. The results demonstrate that our AIC MLLM can efficiently correct failure samples by leveraging interaction experience prompts.Our project website is https://sites.google.com/view/aic-mllm.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
