arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.08626cs.CV

WildDet3D: 在真实世界中扩展可提示的3D检测

WildDet3D: Scaling Promptable 3D Detection in the Wild

Allen Institute for AI(人工智能研究院);University of Washington(华盛顿大学);Cornell University(康奈尔大学);Johns Hopkins University(约翰霍普金斯大学)
Weikai Huang, Jieyu Zhang, Sijun Li, Taoyang Jia, Jiafei Duan, Yunqian Cheng, Jaemin Cho, Matthew Wallingford, Rustin Soraki, Chris Dongjoo Kim, Shuo Liu, Donov… 展开作者

Weikai Huang, Jieyu Zhang, Sijun Li, Taoyang Jia, Jiafei Duan, Yunqian Cheng, Jaemin Cho, Matthew Wallingford, Rustin Soraki, Chris Dongjoo Kim, Shuo Liu, Donovan Clay, Taira Anderson, Winson Han, Ali Farhadi, Bharath Hariharan, Zhongzheng Ren, Ranjay Krishna

AI总结:

本文提出WildDet3D,一种统一的几何感知架构,支持文本、点和框提示,并在推理时整合辅助深度信号。同时,构建了最大的开放3D检测数据集WildDet3D-Data,包含13.5K类别的100万张图像,在多个基准测试中取得新突破。

摘要:

理解单张图像中的3D物体是空间智能的核心。关键步骤是单目3D物体检测--从输入RGB图像中恢复物体的范围、位置和方向。为了在开放世界中实用,此类检测器必须超越封闭集类别,支持多样的提示模态,并利用可用的几何线索。进展受两个瓶颈限制:现有方法针对单一提示类型设计,缺乏整合额外几何线索的机制;当前3D数据集仅覆盖受控环境中的狭窄类别,限制了开放世界迁移。本文解决了这两个缺口。首先,我们引入WildDet3D,一种统一的几何感知架构,原生支持文本、点和框提示,并可在推理时整合辅助深度信号。其次,我们提出WildDet3D-Data,目前最大的开放3D检测数据集,通过从现有2D注释中生成候选3D框并保留仅有人验证的,得到超过100万张图像,涵盖13.5K类别的多样化真实世界场景。WildDet3D在多个基准和设置中建立了新的状态-of-the-art。在开放世界设置中,它在我们新引入的WildDet3D-Bench上使用文本和框提示达到22.6/24.8 AP3D。在Omni3D上,它分别达到34.2/36.4 AP3D。在零样本评估中,它在Argoverse 2和ScanNet上分别达到40.3/48.9 ODS。值得注意的是,在推理时整合深度线索带来了显著的额外收益(平均+20.7 AP在多个设置中)。

英文摘要:

Understanding objects in 3D from a single image is a cornerstone of spatial intelligence. A key step toward this goal is monocular 3D object detection--recovering the extent, location, and orientation of objects from an input RGB image. To be practical in the open world, such a detector must generalize beyond closed-set categories, support diverse prompt modalities, and leverage geometric cues when available. Progress is hampered by two bottlenecks: existing methods are designed for a single prompt type and lack a mechanism to incorporate additional geometric cues, and current 3D datasets cover only narrow categories in controlled environments, limiting open-world transfer. In this work we address both gaps. First, we introduce WildDet3D, a unified geometry-aware architecture that natively accepts text, point, and box prompts and can incorporate auxiliary depth signals at inference time. Second, we present WildDet3D-Data, the largest open 3D detection dataset to date, constructed by generating candidate 3D boxes from existing 2D annotations and retaining only human-verified ones, yielding over 1M images across 13.5K categories in diverse real-world scenes. WildDet3D establishes a new state-of-the-art across multiple benchmarks and settings. In the open-world setting, it achieves 22.6/24.8 AP3D on our newly introduced WildDet3D-Bench with text and box prompts. On Omni3D, it reaches 34.2/36.4 AP3D with text and box prompts, respectively. In zero-shot evaluation, it achieves 40.3/48.9 ODS on Argoverse 2 and ScanNet. Notably, incorporating depth cues at inference time yields substantial additional gains (+20.7 AP on average across settings).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑