Large foundation models have recently emerged as a prominent focus of interest, attaining superior performance in widespread scenarios. Due to the scarcity of 3D data, many efforts have been made to adapt pre-trained transformers from vision to 3D domains. However, such 2D-to-3D approaches are still limited, due to the potential loss of spatial geometries and high computation cost. More importantly, their frameworks are mainly designed for 2D models, lacking a general any-to-3D paradigm. In this paper, we introduce Any2Point, a parameter-efficient method to empower any-modality large models (vision, language, audio) for 3D understanding. Given a frozen transformer from any source modality, we propose a 3D-to-any (1D or 2D) virtual projection strategy that correlates the input 3D points to the original 1D or 2D positions within the source modality. This mechanism enables us to assign each 3D token with a positional encoding paired with the pre-trained model, which avoids 3D geometry loss caused by the true projection and better motivates the transformer for 3D learning with 1D/2D positional priors. Then, within each transformer block, we insert an any-to-3D guided adapter module for parameter-efficient fine-tuning. The adapter incorporates prior spatial knowledge from the source modality to guide the local feature aggregation of 3D tokens, compelling the semantic adaption of any-modality transformers. We conduct extensive experiments to showcase the effectiveness and efficiency of our method. Code and models are released at https://github.com/Ivan-Tang-3D/Any2Point.
Empowering Time Series Analysis with Foundation Models: A Comprehensive Survey
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));The Hong Kong University of Science and Technology(香港科学与技术大学);Shanghai University of Finance and Economics(上海财经大学)
3DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding
School of Geospatial Engineering and Science(测绘工程与科学学院);Sun Yat-sen University(中山大学);College of Electronic Science(电子科学学院);National University of Defense Technology(国防科技大学);Department of Civil and Environmental Engineering(土木与环境工程系);Norwegian University of Science and Technology(挪威科技大学);National Geomatics Center of China(中国国家测绘地理信息中心)
ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability
Nankai University(南开大学);Shanghai Innovation Institute(上海创新研究院);University of Science and Technology of China(中国科学技术大学);Wuhan University(武汉大学);Shanghai AI Laboratory(上海人工智能实验室)
SVL: Empowering Spiking Neural Networks for Efficient 3D Open-World Understanding
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Future Technology, University of Chinese Academy of Sciences(中国科学院大学未来技术学院);Zhongguancun Academy(中关村学院);University of Science and Technology of China(中国科学技术大学);Peking University(北京大学)
The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation
The University of Texas at Austin(德克萨斯大学奥斯汀分校);Massachusetts Institute of Technology(麻省理工学院);South China University of Technology(华南理工大学);Tsinghua University(清华大学);Shanghai Qi Zhi Institute(上海期智研究院)
From Consistency to Complementarity: Aligned and Disentangled Multi-modal Learning for Time Series Understanding and Reasoning
The Hong Kong University of Science(香港科学与技术大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学)
Traj-LLM: A New Exploration for Empowering Trajectory Prediction with Pre-trained Large Language Models
Beihang University(北京航空航天大学);Research Institute for Frontier Science, Beihang University(北京航空航天大学前沿科学研究院);Beijing University of Technology(北京工业大学);College of Metropolitan Transportation, Beijing University of Technology(北京工业大学城市交通学院)
RGBX-R1: Visual Modality Chain-of-Thought Guided Reinforcement Learning for Multimodal Grounding
School of Artificial Intelligence, Tianjin University(天津大学人工智能学院);Low-Altitude Intelligence Lab, Xiong’an National Innovation Center(雄安国家创新中心低空智能实验室);Xiong’an Guochuang Lantian Technology Co., Ltd.(雄安国创莲田科技有限公司);College of Electronic Science and Technology, National University of Defense Technology(国防科技大学电子科学学院)
Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding
Zhejiang Gongshang University(浙江工商大学);Peking University(北京大学);Zhejiang Key Lab of E-Commerce(浙江省电子商务重点实验室);Huazhong University of Science and Technology(华中科技大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Science and Technology of China(中国科学技术大学);Hefei University of Technology(合肥工业大学)
Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models
CVIU Lab, University of Arkansas, USA(美国亚拉巴马大学CVIU实验室);Vietnam National University, Ho Chi Minh City University of Science, Vietnam(越南国家大学胡志明市科技大学);Carnegie Mellon University, USA(美国卡内基梅隆大学)