LL3DA：面向全3D理解、推理与规划的视觉交互式指令微调
LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning
AI总结：
本文提出LL3DA，一种直接以点云为输入的大型语言3D助手，通过融合文本指令与视觉提示提升复杂3D场景中的理解、推理与规划能力，并在3D密集字幕和3D问答任务上超越现有模型。
摘要：
大型多模态模型（LMM）的最新进展使人机交互中的各种应用成为可能。然而，开发能够在复杂多样的3D环境中进行理解、推理和规划的LMM仍然是一个具有挑战性的课题，尤其是考虑到需要理解3D场景的置换不变点云3D表示。现有工作借助多视图图像，并将2D特征投影到3D空间作为3D场景表示。然而，这导致了巨大的计算开销和性能下降。在本文中，我们提出了LL3DA，一个大型语言3D助手，它以点云作为直接输入，并响应文本指令和视觉提示。这有助于LMM更好地理解人类交互，并进一步帮助消除杂乱3D场景中的歧义。实验表明，LL3DA取得了显著的结果，并在3D密集字幕和3D问答上超越了各种3D视觉-语言模型。
英文摘要：
Recent advances in Large Multimodal Models (LMM) have made it possible for various applications in human-machine interactions. However, developing LMMs that can comprehend, reason, and plan in complex and diverse 3D environments remains a challenging topic, especially considering the demand for understanding permutation-invariant point cloud 3D representations of the 3D scene. Existing works seek help from multi-view images, and project 2D features to 3D space as 3D scene representations. This, however, leads to huge computational overhead and performance degradation. In this paper, we present LL3DA, a Large Language 3D Assistant that takes point cloud as direct input and respond to both textual-instructions and visual-prompts. This help LMMs better comprehend human interactions and further help to remove the ambiguities in cluttered 3D scenes. Experiments show that LL3DA achieves remarkable results, and surpasses various 3D vision-language models on both 3D Dense Captioning and 3D Question Answering.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
