MASER: 面向具身3D空间智能的模态自适应专家路由
MASER: Modality-Adaptive Specialist Routing for Embodied 3D Spatial Intelligence
AI总结：
提出MASER框架，通过训练共享VLM骨干的五个模态适配器并学习基于问题选择最佳适配器的神经路由策略，解决具身代理在3D环境中多模态推理时忽略问题语义的问题。
摘要：
在3D环境中，具身代理通过推理自然语言、RGB图像、点云、深度图和相机位姿等多模态信息来回答空间相关问题。现有的视觉语言模型（VLM）在单一模态上微调，完全忽略了可能偏好不同于微调模态的问题语义。为解决这一问题，我们提出MASER（模态自适应专家路由），一个轻量级框架，训练共享VLM骨干的五个不同模态适配器，并学习一个神经路由策略，在推理时根据问题选择最佳适配器。我们使用冻结的句子变换器对每个问题进行编码，并将嵌入通过一个小型多层感知器（MLP），该感知器在oracle适配器-准确率标签上训练。我们在Open3D-VQA基准上评估我们的方法，评估结果表明没有单一模态是普遍最优的——点云答案在51.5%的情况下最佳。MASER以51.3%的oracle一致性进行路由，优于随机森林消融（43.5%），且每个问题仅调用一次适配器。
英文摘要：
In 3D environments, Embodied Agents answer spatially relevant questions through reasoning from a mixture of modalities including natural language, RGB images, point clouds, depth maps and camera poses. Existing Vision-Language models (VLMs) are fine-tuned over a single modality. This completely ignores the question semantics which may favor a different modality than the finetuned modality. To address this, we propose MASER (Modality-Adaptive SpEcialist Routing), a lightweight framework that trains five different modality adapters of a shared VLM backbone and learns a neural routing policy that selects the best adapter based on the question during inference. We encode each question with a frozen sentence transformer and pass the embedding through a small Multi-layer Perceptron (MLP) trained on oracle adapter-accuracy labels. We evaluate our methodology over the Open3D-VQA benchmark and our evaluations show that no single modality is universally optimal -- point-cloud answers are best in 51.5% of cases. MASER routes with 51.3% oracle agreement, outperforming a Random-Forest ablation (43.5%), with only a single adapter call per question.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
