将神经符号程序蒸馏到3D多模态大语言模型中
Distilling Neuro-Symbolic Programs into 3D Multi-modal LLMs
- University of California, Berkeley（加州大学伯克利分校）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
提出APEIRIA，通过三阶段课程学习将符号推理模式蒸馏到3D多模态大语言模型中，实现透明推理与开放词汇空间推理的统一。
AI中文摘要：
当前的3D空间推理方法面临根本性权衡：神经符号3D（NS3D）概念学习器通过组合程序实现可解释推理，但受限于封闭集概念词汇和简单程序；端到端3D多模态大语言模型（3D MLLMs）能处理复杂自然语言和开放词汇概念，但缺乏显式空间验证的黑箱推理。我们提出APEIRIA，一种神经符号3D MLLM，通过将符号推理模式以自然语言思维链形式蒸馏到MLLMs中，桥接两种范式。我们的三阶段课程逐步构建推理能力：a) 3D感知对齐将物体视觉-几何特征接地到LLM，b) CoT-SFT从符号程序轨迹中教授查询分解和逐步验证，c) CoT-RL将推理模式扩展到开放集概念和深度嵌套指令。通过迁移推理模式而非概念特定知识，APEIRIA保留了NS3D的关键优点：透明推理以及规划和感知组件的模块化可互换性。在接地、问答和描述任务上的评估表明，APEIRIA超越了先前的NS3D方法，并在3D空间推理数据集上匹配最先进的3D MLLMs，统一了符号方法的系统推理与MLLMs的灵活性。代码见https://github.com/oceanflowlab/APEIRIA。
英文摘要：
Current 3D spatial reasoning methods face a fundamental trade-off: neuro-symbolic 3D (NS3D) concept learners achieve interpretable reasoning through compositional programs but are constrained to closed-set concept vocabularies and simple programs; end-to-end 3D multi-modal LLMs (3D MLLMs) could handle complex natural language and open-vocabulary concepts but suffer from black-box reasoning without explicit spatial verification. We introduce APEIRIA, a neuro-symbolic 3D MLLM to bridge two paradigms by distilling symbolic reasoning patterns into MLLMs with natural language chain-of-thought. Our three-stage curriculum progressively builds reasoning capabilities: a) 3D perception alignment grounds object visual-geometric features to the LLM, b) CoT-SFT teaches query decomposition and stepwise verification from symbolic program traces, and c) CoT-RL extends reasoning patterns to open-set concepts and deeply nested instructions. By transferring reasoning patterns rather than concept-specific knowledge, APEIRIA preserves key NS3D virtues: transparent reasoning and modular interchangeability of planning and perception components. Evaluations on grounding, question answering, and captioning show that APEIRIA surpasses prior NS3D methods and matches state-of-the-art 3D MLLMs on 3D spatial reasoning datasets, unifying symbolic methods' systematic reasoning with MLLMs' flexibility. Code is available at https://github.com/oceanflowlab/APEIRIA.
