通知：联邦节假日调休：哥伦布日 / 原住民日（Columbus Day / Indigenous Peoples' Day），今日（13日）无更新。通知：联邦节假日调休：哥伦布日 / 原住民日（Columbus Day / Indigenous Peoples' Day），今日（13日）无更新。
arXiv 2503.13185cs.CVcs.AI
3DAxisPrompt：提升 GPT-4o 的 3D 定位与推理能力
3DAxisPrompt: Promoting the 3D Grounding and Reasoning in GPT-4o
Shanghai AI Lab（上海人工智能实验室）；Dalian University of Technology（大连理工大学）；School of Geodesy and Geomatics, Wuhan University（武汉大学测绘学院）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）
提出 3DAxisPrompt，通过 3D 坐标轴与 SAM 掩码为 GPT-4o 提供几何先验，并在四个数据集上验证其 3D 定位与推理能力。
摘要：
多模态大语言模型（MLLMs）在多种任务中展现出令人瞩目的能力，尤其是在配备精心设计的视觉提示时。然而，现有研究主要关注逻辑推理和视觉理解，而 MLLMs 在 3D 视觉中有效运作的能力仍是一个持续探索的领域。本文提出一种名为 3DAxisPrompt 的新型视觉提示方法，以激发 MLLMs 在真实世界场景中的 3D 理解能力。更具体地说，该方法利用 3D 坐标轴和由 Segment Anything Model（SAM）生成的掩码，为 MLLMs 提供显式几何先验，进而将其出色的 2D 定位与推理能力扩展到真实世界的 3D 场景。此外，我们首次全面考察了潜在的视觉提示形式，并总结发现，以揭示 GPT-4o 作为 MLLMs 代表在 3D 理解能力方面的潜力与局限。最后，我们使用四个数据集构建评估环境，即 ScanRefer、ScanNet、FMB 和 nuScene 数据集，覆盖多种 3D 任务。在此基础上，我们进行了大量定量与定性实验，结果证明了所提方法的有效性。总体而言，研究表明，在 3DAxisPrompt 的帮助下，MLLMs 能够有效感知真实世界场景中物体的 3D 位置。尽管如此，单一的提示工程方法并不能在所有 3D 任务上始终取得最佳结果。本研究凸显了利用提示工程技术让 MLLMs 完成 3D 视觉定位/推理的可行性。
英文摘要：
Multimodal Large Language Models (MLLMs) exhibit impressive capabilities across a variety of tasks, especially when equipped with carefully designed visual prompts. However, existing studies primarily focus on logical reasoning and visual understanding, while the capability of MLLMs to operate effectively in 3D vision remains an ongoing area of exploration. In this paper, we introduce a novel visual prompting method, called 3DAxisPrompt, to elicit the 3D understanding capabilities of MLLMs in real-world scenes. More specifically, our method leverages the 3D coordinate axis and masks generated from the Segment Anything Model (SAM) to provide explicit geometric priors to MLLMs and then extend their impressive 2D grounding and reasoning ability to real-world 3D scenarios. Besides, we first provide a thorough investigation of the potential visual prompting formats and conclude our findings to reveal the potential and limits of 3D understanding capabilities in GPT-4o, as a representative of MLLMs. Finally, we build evaluation environments with four datasets, i.e., ScanRefer, ScanNet, FMB, and nuScene datasets, covering various 3D tasks. Based on this, we conduct extensive quantitative and qualitative experiments, which demonstrate the effectiveness of the proposed method. Overall, our study reveals that MLLMs, with the help of 3DAxisPrompt, can effectively perceive an object's 3D position in real-world scenarios. Nevertheless, a single prompt engineering approach does not consistently achieve the best outcomes for all 3D tasks. This study highlights the feasibility of leveraging MLLMs for 3D vision grounding/reasoning with prompt engineering techniques.
A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysis
The University of Sydney（悉尼大学）；Tencent AI Lab, Tencent（腾讯AI实验室）；Guangzhou University of Chinese Medicine（广州中医药大学）；The University of Wollongong（伍伦贡大学）；The University of Adelaide（阿德莱德大学）
PG-Video-LLaVA: Pixel Grounding Large Video-Language Models
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Australian National University（澳大利亚国立大学）；University of Central Florida（中佛罗里达大学）；Linköping University（林雪平大学）
Towards Unified Surgical Scene Understanding:Bridging Reasoning and Grounding via MLLMs
Southern University of Science and Technology（南方科技大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；Northwestern University（西北大学）；University of Alberta（阿尔伯塔大学）；Yale University（耶鲁大学）；Nanfang Hospital（南华医院）；Shenzhen University of Advanced Technology（深圳大学先进技术研究院）
VisDoT : Enhancing Visual Reasoning through Human-Like Interpretation Grounding and Decomposition of Thought
Department of Computer Science and Artificial Intelligence, Dongguk University（东国大学计算机科学与人工智能系）；Department of Electronics and Electrical Engineering, Dongguk University（东国大学电子与电气工程系）
Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models
New York University, New York, USA（纽约大学）；Tsinghua University, Beijing, China（清华大学）；Columbia University, New York, USA（哥伦比亚大学）；University of Michigan, Ann Arbor, USA（密歇根大学）
VTG-GPT: Tuning-Free Zero-Shot Video Temporal Grounding with GPT
School of Electronic Science and Engineering, Nanjing University（南京大学电子科学与工程学院）；School of Physics and Electronics, Hubei Normal University（湖北师范大学物理与电子科学学院）
