重审3D LLM基准:我们真的在测试3D能力吗?
Revisiting 3D LLM Benchmarks: Are We Really Testing 3D Capabilities?
AI总结:
该研究揭示3D LLM基准存在“2D作弊”,通过VLM跨基准测试证明任务可能仅依赖渲染图像,并提出区分3D与1D/2D能力的真实3D理解评估原则。
摘要:
在本工作中,我们识别出3D LLM评估中的“2D作弊”问题:这些任务可能被VLM利用点云的渲染图像轻松解决,暴露出对3D LLM独有3D能力的评估是无效的。我们在多个3D LLM基准上测试VLM性能,并以此为参照,提出更好评估真正3D理解的原则。我们还主张在评估3D LLM时,将3D能力与1D或2D方面明确区分。代码和数据可在https://github.com/LLM-class-group/Revisiting-3D-LLM-Benchmarks获取。
英文摘要:
In this work, we identify the "2D-Cheating" problem in 3D LLM evaluation, where these tasks might be easily solved by VLMs with rendered images of point clouds, exposing ineffective evaluation of 3D LLMs' unique 3D capabilities. We test VLM performance across multiple 3D LLM benchmarks and, using this as a reference, propose principles for better assessing genuine 3D understanding. We also advocate explicitly separating 3D abilities from 1D or 2D aspects when evaluating 3D LLMs. Code and data are available at https://github.com/LLM-class-group/Revisiting-3D-LLM-Benchmarks
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。