ReSemAct：通过语义结构化和效用细化推进细粒度机器人操作
ReSemAct: Advancing Fine-Grained Robotic Manipulation via Semantic Structuring and Affordance Refinement
AI总结：
ReSemAct 通过语义结构化和效用细化方法，在细粒度机器人操作中实现更精确的效用目标生成与动态环境适应。
摘要：
细粒度机器人操作需要将自然语言 grounding 到适当的效用目标。然而，大多数由基础模型驱动的方法往往将丰富的语义压缩成过于简化的效用，阻碍了隐式语义信息的利用。为了解决这些挑战，我们提出了 ReSemAct，一种新的统一操作框架，引入了语义结构化和效用细化（SSAR），由多模态大语言模型（MLLMs）和视觉基础模型（VFMs）之间的自动化协同推理驱动。具体而言，语义结构化模块从自然语言和 RGB 观察中推导出统一的语义效用描述，将效用区域、隐含功能意图和粗效用锚点组织成结构化的表示，以供后续细化。在此规范基础上，效用细化策略实例化了两个互补的流程，分别专注于几何和位置，产生细粒度的效用目标。这些细化的目标随后被编码为实时的关节空间优化目标，使在动态环境中能够实现反应性和鲁棒性的操作。进行了大量的仿真和现实世界实验，在语义丰富的家庭和稀疏的化学实验室环境中。结果表明，ReSemAct 在零样本条件下能够执行多样任务，展示了在细粒度操作中基础模型的 SSAR 的鲁棒性。代码和视频见 https://github.com/scy-v/ReSemAct 和 https://resemact.github.io。
英文摘要：
Fine-grained robotic manipulation requires grounding natural language into appropriate affordance targets. However, most existing methods driven by foundation models often compress rich semantics into oversimplified affordances, preventing exploitation of implicit semantic information. To address these challenges, we present ReSemAct, a novel unified manipulation framework that introduces Semantic Structuring and Affordance Refinement (SSAR), powered by the automated synergistic reasoning between Multimodal Large Language Models (MLLMs) and Vision Foundation Models (VFMs). Specifically, the Semantic Structuring module derives a unified semantic affordance description from natural language and RGB observations, organizing affordance regions, implicit functional intent, and coarse affordance anchors into a structured representation for downstream refinement. Building upon this specification, the Affordance Refinement strategy instantiates two complementary flows that separately specialize geometry and position, yielding fine-grained affordance targets. These refined targets are then encoded as real-time joint-space optimization objectives, enabling reactive and robust manipulation in dynamic environments. Extensive simulation and real-world experiments are conducted in semantically rich household and sparse chemical lab environments. The results demonstrate that ReSemAct performs diverse tasks under zero-shot conditions, showcasing the robustness of SSAR with foundation models in fine-grained manipulation. Code and videos at https://github.com/scy-v/ReSemAct and https://resemact.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
