3D-GRAND：一个具备更优接地能力与更少幻觉的百万级3D-LLM数据集
3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination
AI总结：
本研究推出百万级3D-LLM数据集3D-GRAND及幻觉评估基准3D-POPE，验证了指令微调可提升模型接地能力、减少幻觉，证实数据集规模的缩放效应与模拟到真实迁移的可行性，为具身AI研究提供支撑。
摘要：
语言与3D感知的融合对于能够理解物理世界并与之交互的具身智能体和机器人至关重要。尽管大语言模型（LLM）已展现出令人瞩目的语言理解和生成能力，但其向3D环境的适配（即3D-LLM）仍处于早期阶段，核心挑战在于缺乏语言与3D场景间存在密集接地关联的大规模数据集。我们推出了3D-GRAND这一开创性的大规模数据集，包含40087个家居场景及与之配对的620万条密集接地场景语言指令。实验结果表明，使用3D-GRAND进行指令微调可显著提升3D-LLM的接地能力并减少幻觉。作为研究贡献的一部分，我们提出了综合性基准3D-POPE，用于系统评估3D-LLM的幻觉问题，以实现模型间的公平比较。实验还揭示了数据集规模与3D-LLM性能之间的缩放效应，凸显了大规模3D-文本数据集对具身AI研究的重要性。研究结果展现了有效的模拟到真实迁移的初步迹象，表明在大规模合成数据上训练的模型可在真实世界3D扫描数据上取得良好表现。我们希望通过3D-GRAND和3D-POPE为具身AI社区提供资源与洞见，推动研发出更可靠、接地能力更强的3D-LLM。项目官网：https://3d-grand.github.io
英文摘要：
The integration of language and 3D perception is crucial for embodied agents and robots that comprehend and interact with the physical world. While large language models (LLMs) have demonstrated impressive language understanding and generation capabilities, their adaptation to 3D environments (3D-LLMs) remains in its early stages. A primary challenge is a lack of large-scale datasets with dense grounding between language and 3D scenes. We introduce 3D-GRAND, a pioneering large-scale dataset comprising 40,087 household scenes paired with 6.2 million densely-grounded scene-language instructions. Our results show that instruction tuning with 3D-GRAND significantly enhances grounding capabilities and reduces hallucinations in 3D-LLMs. As part of our contributions, we propose a comprehensive benchmark 3D-POPE to systematically evaluate hallucination in 3D-LLMs, enabling fair comparisons of models. Our experiments highlight a scaling effect between dataset size and 3D-LLM performance, emphasizing the importance of large-scale 3D-text datasets for embodied AI research. Our results demonstrate early signals for effective sim-to-real transfer, indicating that models trained on large synthetic data can perform well on real-world 3D scans. Through 3D-GRAND and 3D-POPE, we aim to equip the embodied AI community with resources and insights to lead to more reliable and better-grounded 3D-LLMs. Project website: https://3d-grand.github.io
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
