Hyperbolic embeddings have demonstrated their effectiveness in capturing measures of uncertainty and hierarchical relationships across various deep-learning tasks, including image segmentation and active learning. However, their application in modern vision-language models (VLMs) has been limited. A notable exception is MERU, which leverages the hierarchical properties of hyperbolic space in the CLIP ViT-large model, consisting of hundreds of millions parameters. In our work, we address the challenges of scaling multi-modal hyperbolic models by orders of magnitude in terms of parameters (billions) and training complexity using the BLIP-2 architecture. Although hyperbolic embeddings offer potential insights into uncertainty not present in Euclidean embeddings, our analysis reveals that scaling these models is particularly difficult. We propose a novel training strategy for a hyperbolic version of BLIP-2, which allows to achieve comparable performance to its Euclidean counterpart, while maintaining stability throughout the training process and showing a meaningful indication of uncertainty with each embedding.
Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?
LMU Munich（慕尼黑大学）；Siemens AG（西门子公司）；Technical University of Munich（慕尼黑工业大学）；University of Oxford（牛津大学）；University of California, Santa Barbara（加州大学圣巴巴拉分校）
Department of Data Science, New Jersey Institute of Technology（数据科学系，新泽西理工学院）；Department of Biomedical Informatics, Stony Brook University（生物医学信息学系，石溪大学）
MLaGA: Multimodal Large Language and Graph Assistant
New York University（纽约大学）；New York University Shanghai（纽约大学上海）；New York University Brooklyn（纽约大学布鲁克林）；Virginia Polytechnic Institute and State University（弗吉尼亚理工大学）；New York University Abu Dhabi（纽约大学阿布扎克）
Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark
Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Pattern Recognition Center, WeChat AI, Tencent Inc, China（腾讯人工智能研究院）；Kennesaw State University（凯斯西储大学）
Hong Kong University of Science and Technology（香港科技大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
The recent large-scale vision-language pre-training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on various multimodal alignment tasks. Despite its success, the resulting models are not capable of multimodal generative tasks due to the weak text encoder. To tackle this problem, we propose to augment the dual-stream VLP model with a textual pre-trained language model (PLM) via vision-language knowledge distillation (VLKD), enabling the capability for multimodal generation. VLKD is pretty data- and computation-efficie…
We introduce a novel framework for evaluating multimodal deep learning models with respect to their language understanding and generalization abilities. In this approach, artificial data is automatically generated according to the experimenter's specifications. The content of the data, both during training and evaluation, can be controlled in detail, which enables tasks to be created that require true generalization abilities, in particular the combination of previously introduced concepts in novel ways. We demonstrate the potential of our methodology by evaluating various visual question answ…
Visual-Oriented Fine-Grained Knowledge Editing for MultiModal Large Language Models
Hefei University of Technology（合肥工业大学）；University of Science and Technology of China（中国科学技术大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）
MMUnlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；Tongji University（同济大学）
Align is not Enough: Multimodal Universal Jailbreak Attack against Multimodal Large Language Models
School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机科学与信息工程学院）；Tsinghua University（清华大学）；Institute of automation, Chinese academy of science（中国科学院自动化研究所）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）
NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMs
College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）；Zhongguancun Academy（中关村学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Tianjin University（天津大学）；Nankai University（南开大学）
Infi-MMR: Curriculum-based Unlocking Multimodal Reasoning via Phased Reinforcement Learning in Multimodal Small Language Models
The Hong Kong Polytechnic University（香港理工大学）；Zhejiang University（浙江大学）；University of Electronic Science and Technology of China（电子科技大学）；Reallm Labs（Reallm 实验室）；Amazon（亚马逊）；The Hong Kong University of Science and Technology（香港理工大学）
mChartQA: A universal benchmark for multimodal Chart Question Answer based on Vision-Language Alignment and Reasoning
Shenyang Institute of Computing Technology, Chinese Academy of Sciences（中国科学院沈阳计算技术研究所）；Beijing Wenge Technology Co., Ltd（北京文格科技有限公司）；University of Chinese Academy of Sciences（中国科学院大学）
InfiR : Crafting Effective Small Language Models and Multimodal Small Language Models in Reasoning
Reallm Labs（Reallm实验室）；The Hong Kong Polytechnic University（香港理工大学）；Zhejiang University（浙江大学）；South China University of Technology（华南理工大学）；Harbin Institute of Technology（哈尔滨工业大学）；Dalian University of Technology（大连理工大学）；University of Electronic Science and Technology of China（电子科技大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；TikTok（字节跳动旗下TikTok）；Amazon（亚马逊）
