In this work, we present Uni3DL, a unified model for 3D and Language understanding. Distinct from existing unified vision-language models in 3D which are limited in task variety and predominantly dependent on projected multi-view images, Uni3DL operates directly on point clouds. This approach significantly expands the range of supported tasks in 3D, encompassing both vision and vision-language tasks in 3D. At the core of Uni3DL, a query transformer is designed to learn task-agnostic semantic and mask outputs by attending to 3D visual features, and a task router is employed to selectively generate task-specific outputs required for diverse tasks. With a unified architecture, our Uni3DL model enjoys seamless task decomposition and substantial parameter sharing across tasks. Uni3DL has been rigorously evaluated across diverse 3D vision-language understanding tasks, including semantic segmentation, object detection, instance segmentation, visual grounding, 3D captioning, and text-3D cross-modal retrieval. It demonstrates performance on par with or surpassing state-of-the-art (SOTA) task-specific models. We hope our benchmark and Uni3DL model will serve as a solid step to ease future research in unified models in the realm of 3D and language understanding. Project page: https://uni3dl.github.io.
Fuse & Calibrate: A bi-directional Vision-Language Guided Framework for Referring Image Segmentation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院)
VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework
Stanford University(斯坦福大学);Peking University(北京大学);Harvard University(哈佛大学);University of Washington, Seattle(华盛顿大学西雅图分校);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)
SpatialBot: Precise Spatial Understanding with Vision Language Models
School of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学人工智能学院);Stanford University(斯坦福大学);BAAI(北京智源人工智能研究院);Peking University(北京大学);University of Oxford(牛津大学);Southeast University(东南大学)
Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Model
School of Software Technology, Zhejiang University(浙江大学软件学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);School of Electronic and Information, Shanghai Dianji University(上海电机学院电子信息学院)
Using Left and Right Brains Together: Towards Vision and Language Planning
Southern University of Science and Technology(南方科技大学);Microsoft Research Asia(微软亚洲研究院);The Hong Kong University of Science and Technology(香港科技大学);Xi’an Jiaotong University(西安交通大学);City University of Hong Kong(香港城市大学)
Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机与软件学院);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Research Center of Artificial Intelligence, Peng Cheng Laboratory(鹏城实验室人工智能研究中心)
LM4LV: A Frozen Large Language Model for Low-level Vision Tasks
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Nanjing University(南京大学);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
MOVE: A Mixture-of-Vision-Encoders Approach for Domain-Focused Vision-Language Processing
Skolkovo Institute of Science and Technology(斯科尔科沃科学技术研究院);Artificial Intelligence Research Institute (AIRI)(人工智能研究院);Higher School of Economics(高等经济学院)
Griffon-G: Bridging Vision-Language and Vision-Centric Tasks via Large Multimodal Models
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所基础模型研究中心);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Peng Cheng Laboratory(鹏城实验室);Wuhan AI Research(武汉人工智能研究院)
VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks
OpenGVLab(开放通用视觉实验室);Shanghai AI Laboratory(上海人工智能实验室);Tsinghua University(清华大学);SenseTime Research(商汤科技研究院);Nanjing University(南京大学);The University of Hong Kong(香港大学);Beijing Institute of Technology(北京理工大学);The Hong Kong University of Science and Technology(香港科技大学);The Chinese University of Hong Kong(香港中文大学)
VisionLLM v2 提出端到端通用多模态大模型,通过 super link 机制连接 MLLM 与任务专用解码器,并在数百个视觉语言任务上联合训练,以共享参数实现目标定位、姿态估计、图像生成编辑等多样化任务,性能可比肩专用模型。
HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding
Tsinghua University(清华大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Johns Hopkins University(约翰斯·霍普金斯大学);SenseTime Research(商汤科技研究院);Nanjing University(南京大学)
Towards Mitigating Modality Bias in Vision-Language Models for Temporal Action Localization
UVLab, Department of Computer Science, University of Warwick(华威大学计算机科学系UVLab);Department of Automation, University of Cambridge(剑桥大学自动化系);Department of Computer Science, The University of Sheffield(谢菲尔德大学计算机科学系)