Agent3D-Zero: An Agent for Zero-shot 3D Understanding
University of Science and Technology of China(中国科学技术大学);Shanghai Artificial Intelligent Laboratory(上海人工智能实验室);The University of Sydney(悉尼大学);The University of Adelaide(阿德莱德大学);Australian Institute for Machine Learning(澳大利亚机器学习研究所)
The ability to understand and reason the 3D real world is a crucial milestone towards artificial general intelligence. The current common practice is to finetune Large Language Models (LLMs) with 3D data and texts to enable 3D understanding. Despite their effectiveness, these approaches are inherently limited by the scale and diversity of the available 3D data. Alternatively, in this work, we introduce Agent3D-Zero, an innovative 3D-aware agent framework addressing the 3D scene understanding in a zero-shot manner. The essence of our approach centers on reconceptualizing the challenge of 3D scene perception as a process of understanding and synthesizing insights from multiple images, inspired by how our human beings attempt to understand 3D scenes. By consolidating this idea, we propose a novel way to make use of a Large Visual Language Model (VLM) via actively selecting and analyzing a series of viewpoints for 3D understanding. Specifically, given an input 3D scene, Agent3D-Zero first processes a bird's-eye view image with custom-designed visual prompts, then iteratively chooses the next viewpoints to observe and summarize the underlying knowledge. A distinctive advantage of Agent3D-Zero is the introduction of novel visual prompts, which significantly unleash the VLMs' ability to identify the most informative viewpoints and thus facilitate observing 3D scenes. Extensive experiments demonstrate the effectiveness of the proposed framework in understanding diverse and previously unseen 3D environments.
SpatialNav: Leveraging Spatial Scene Graphs for Zero-Shot Vision-and-Language Navigation
Fudan University, Shanghai, China(复旦大学);University of Southern California, Los Angeles, USA(南加州大学);Australian Institute for Machine Learning, Adelaide University, Australia(澳大利亚机器学习研究所);Shanghai Innovation Institute, Shanghai, China(上海创新研究院)
Shanghai Jiao Tong University(上海交通大学);Xinjiang University of Finance and Economics(新疆财经大学);Shenzhen University(深圳大学);Southern University of Science and Technology(南方科技大学)
Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videos
Bytedance Inc.(字节跳动公司);AAII, UTS(悉尼科技大学人工智能研究院);MBZUAI(穆罕默德·本·扎耶德人工智能大学);University of Science and Technology of China(中国科学技术大学);Data61, CSIRO(澳大利亚联邦科学与工业研究组织数据61部门)
Agentic Collaborative Cognition for Zero-Shot 3D Understanding
Sichuan University(四川大学);Adelaide University(阿德莱德大学);University of Electronic Science and Technology of China(电子科技大学);Beijing Institute of Technology(北京理工大学)
ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models
Tongji University(同济大学);The Chinese University of Hong Kong(香港中文大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);S-Lab, Nanyang Technological University(南洋理工大学S-Lab)
AgMTR: Agent Mining Transformer for Few-shot Segmentation in Remote Sensing
Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院空天信息创新研究院);School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences(中国科学院大学电子电气与通信工程学院);University of Chinese Academy of Sciences(中国科学院大学);Institute of Electronics, Chinese Academy of Sciences(中国科学院电子学研究所);Tsinghua University(清华大学)
VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding
The Chinese University of Hong Kong(香港中文大学);Shanghai AI Laboratory(上海人工智能实验室);Zhejiang University(浙江大学);Centre for Perceptual and Interactive Intelligence(感知与交互智能中心)
Effect of Temporal Resolution on Zero-Shot Semantic Understanding of Rapid Human Actions
Institute of Industrial Science, The University of Tokyo(东京大学工业科学研究所);Interfaculty Initiative in Information Studies, Graduate School of Interdisciplinary Information Studies, The University of Tokyo(东京大学跨学科信息研究 graduate school)
IPFormer-VideoLLM: Enhancing Multi-modal Video Understanding for Multi-shot Scenes
School of AIA, Huazhong University of Science and Technology(华中科技大学人工智能学院);Deepeleph Intelligent Technology(DeepEleph智能技术);JD Explore Academy(JD探索学院);Department of Electronic Engineering, Tsinghua University(清华大学电子工程系)