AC-DiT：面向移动操作的自适应协调扩散Transformer
AC-DiT: Adaptive Coordination Diffusion Transformer for Mobile Manipulation
AI总结：
AC-DiT通过mobility-to-body条件机制和感知感知的多模态融合策略，解决移动基座与机械臂协调及不同阶段感知需求差异问题，实现端到端移动操作。
摘要：
近年来，移动操作因能够在家庭任务中实现语言条件下的机器人控制而受到越来越多的关注。然而，现有方法在协调移动基座与机械臂方面仍面临挑战，主要受限于两个问题。一方面，它们未能显式建模移动基座对机械臂控制的影响，在高自由度情况下容易导致误差累积。另一方面，它们在整个移动操作过程中使用相同的视觉观测模态（例如，要么全部使用2D，要么全部使用3D），忽视了移动操作不同阶段对多模态感知的不同需求。为此，我们提出了自适应协调扩散Transformer（AC-DiT），以增强移动基座与机械臂的协调，实现端到端移动操作。首先，由于移动基座的运动直接影响机械臂的动作，我们引入了一种mobility-to-body条件机制，引导模型先提取基座运动表示，再将其作为预测全身动作的上下文先验。这使得全身控制能够考虑移动基座运动可能带来的影响。其次，为满足移动操作不同阶段的感知需求，我们设计了一种感知感知的多模态条件策略，动态调整多种2D视觉图像与3D点云之间的融合权重，从而生成适应当前感知需求的视觉特征。这使得模型能够，例如，在语义信息对动作预测至关重要时自适应地更多依赖2D输入，而在需要精确空间理解时更加重视3D几何信息。我们通过在仿真和真实世界移动操作任务上的大量实验验证了AC-DiT。
英文摘要：
Recently, mobile manipulation has attracted increasing attention for enabling language-conditioned robotic control in household tasks. However, existing methods still face challenges in coordinating mobile base and manipulator, primarily due to two limitations. On the one hand, they fail to explicitly model the influence of the mobile base on manipulator control, which easily leads to error accumulation under high degrees of freedom. On the other hand, they treat the entire mobile manipulation process with the same visual observation modality (e.g., either all 2D or all 3D), overlooking the distinct multimodal perception requirements at different stages during mobile manipulation. To address this, we propose the Adaptive Coordination Diffusion Transformer (AC-DiT), which enhances mobile base and manipulator coordination for end-to-end mobile manipulation. First, since the motion of the mobile base directly influences the manipulator's actions, we introduce a mobility-to-body conditioning mechanism that guides the model to first extract base motion representations, which are then used as context prior for predicting whole-body actions. This enables whole-body control that accounts for the potential impact of the mobile base's motion. Second, to meet the perception requirements at different stages of mobile manipulation, we design a perception-aware multimodal conditioning strategy that dynamically adjusts the fusion weights between various 2D visual images and 3D point clouds, yielding visual features tailored to the current perceptual needs. This allows the model to, for example, adaptively rely more on 2D inputs when semantic information is crucial for action prediction, while placing greater emphasis on 3D geometric information when precise spatial understanding is required. We validate AC-DiT through extensive experiments on both simulated and real-world mobile manipulation tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
