LocateAnything3D: 基于视觉-语言的3D检测与链式视觉推理
LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sight
AI总结：
LocateAnything3D通过将3D检测转化为下一个token预测问题，实现了多目标3D检测，取得了Omni3D基准测试中的最佳成绩。
摘要：
为了在世界中行动，模型必须命名其所看到的事物并了解其在3D空间中的位置。目前的视觉-语言模型（VLMs）在开放性2D描述和语义 grounding 方面表现出色，但多目标3D检测仍然是VLM工具包中缺失的部分。我们提出了LocateAnything3D，一种原生于VLM的配方，将3D检测转化为下一个token的预测问题。关键在于一个简短且明确的链式视觉（Chain-of-Sight, CoS）序列，这反映了人类如何从图像中推理：首先在2D中找到一个对象，然后推断其距离、大小和姿态。解码器首先以视觉链式思考的方式发射2D检测，然后在简单到复杂的课程中预测3D框：在跨对象层面，近到远的顺序减少了早期的歧义并匹配了以自身为中心的实用性；在每个对象内部，以相机为中心、尺寸和旋转的分解按稳定性与可学习性对信息进行排序。这种原生于VLM的接口在不使用专门头的条件下保留了开放词汇和视觉提示能力。在具有挑战性的Omni3D基准测试中，我们的模型取得了最先进的结果，3D AP为38.90，即使在基线给定真实2D框的情况下，也比之前的最佳结果提高了13.98个绝对值。它还能够泛化到未见过的类别，具有强大的鲁棒性。通过将3D检测转化为一个受控的下一个token问题，LocateAnything3D为模型提供了在3D空间中感知的实用基础。
英文摘要：
To act in the world, a model must name what it sees and know where it is in 3D. Today's vision-language models (VLMs) excel at open-ended 2D description and grounding, yet multi-object 3D detection remains largely missing from the VLM toolbox. We present LocateAnything3D, a VLM-native recipe that casts 3D detection as a next-token prediction problem. The key is a short, explicit Chain-of-Sight (CoS) sequence that mirrors how human reason from images: find an object in 2D, then infer its distance, size, and pose. The decoder first emits 2D detections as a visual chain-of-thought, then predicts 3D boxes under an easy-to-hard curriculum: across objects, a near-to-far order reduces early ambiguity and matches ego-centric utility; within each object, a center-from-camera, dimensions, and rotation factorization ranks information by stability and learnability. This VLM-native interface preserves open-vocabulary and visual-prompting capability without specialized heads. On the challenging Omni3D benchmark, our model achieves state-of-the-art results, with 38.90 AP_3D, surpassing the previous best by +13.98 absolute improvement even when the baseline is given ground-truth 2D boxes. It also generalizes zero-shot to held-out categories with strong robustness. By turning 3D detection into a disciplined next-token problem, LocateAnything3D offers a practical foundation for models to perceive in 3D.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
