Cross-attention is commonly adopted in multimodal large language models (MLLMs) for integrating visual information into the language backbone. However, in applications with large visual inputs, such as video understanding, processing a large number of visual tokens in cross-attention layers leads to high memory demands and often necessitates distributed computation across multiple GPUs. Existing distributed attention mechanisms face significant communication overheads, making cross-attention layers a critical bottleneck for efficient training and inference of MLLMs. To address this, we propose LV-XAttn, a distributed, exact cross-attention mechanism with minimal communication overhead. We observe that in applications involving large visual inputs, the size of the query block is typically much smaller than that of the key-value blocks. Thus, in LV-XAttn we keep the large key-value blocks locally on each GPU and exchange smaller query blocks across GPUs. We also introduce an efficient activation recomputation technique to support longer visual context. We theoretically analyze the communication benefits of LV-XAttn and show that it can achieve speedups for a wide range of models. Our evaluations with Llama 3-V, mPLUG-Owl3 and OpenFlamingo models find that LV-XAttn achieves up to 10.62$\times$ end-to-end speedup compared to existing approaches.
XAttn-BMD: Multimodal Deep Learning with Cross-Attention for Femoral Neck Bone Mineral Density Estimation
School of Electronics and Computer Science, University of Southampton, UK（电子与计算机科学学院，索姆塞特大学，英国）；MRC Lifecourse Epidemiology Centre, University of Southampton, Southampton General Hospital, UK（生命课程流行病学研究中心，索姆塞特大学，南安普顿总医院，英国）；NIHR Southampton Biomedical Research Centre, University of Southampton（南安普顿生物医学研究中心，索姆塞特大学）；University Hospital NHS Foundation Trust, Southampton, UK（南安普顿国家健康服务基金会信托，英国）
Pilot: Building the Federated Multimodal Instruction Tuning Framework
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Pengcheng Laboratory（鹏城实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
Large Language Model-Driven Distributed Integrated Multimodal Sensing and Semantic Communications
State Key Laboratory of Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；School of Intelligent Software and Engineering, Nanjing University (Suzhou Campus)（智能软件与工程学院，南京大学（苏州校区））
Integration of Mixture of Experts and Multimodal Generative AI in Internet of Vehicles: A Survey
Nanyang Technological University（南洋理工大学）；Guangdong University of Technology（广东工业大学）；Singapore University of Technology and Design（新加坡科技设计大学）；University of Sydney（悉尼大学）；City University of Hong Kong（香港城市大学）；Sungkyunkwan University（成均馆大学）；University of Waterloo（滑铁卢大学）
Singapore University of Technology and Design（新加坡科技设计大学）
The complex world around us is inherently multimodal and sequential (continuous). Information is scattered across different modalities and requires multiple continuous sensors to be captured. As machine learning leaps towards better generalization to real world, multimodal sequential learning becomes a fundamental research area. Arguably, modeling arbitrarily distributed spatio-temporal dynamics within and across modalities is the biggest challenge in this research area. In this paper, we present a new transformer model, called the Factorized Multimodal Transformer (FMT) for multimodal sequent…
FedNano: Toward Lightweight Federated Tuning for Pretrained Multimodal Large Language Models
LMU Munich（慕尼黑莱布尼茨大学）；Technical University of Munich（慕尼黑技术大学）；Siemens Technology（西门子技术）；Munich Center for Machine Learning（慕尼黑机器学习中心）；University Heidelberg（海德堡大学）
A Step Toward Federated Pretraining of Multimodal Large Language Models
MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统实验室）；King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Pengcheng Laboratory（鹏城实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）
ERGeoBench:A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models
Beijing University of Posts and Telecommunications（北京邮电大学）；State Key Laboratory of Networking and Switching Technology（网络与交换技术国家重点实验室）；School of Materials Science and Engineering（材料科学与工程学院）；China Mobile Research Institute（中国移动研究院）；College of Computing and Data Science（计算与数据科学学院）
École Polytechnique Fédérale de Lausanne (EPFL), Switzerland（瑞士联邦理工学院洛桑校区）；ETH Zürich, Switzerland（瑞士苏黎世联邦理工学院）；T.H. Chan School of Public Health, Harvard University, USA（哈佛大学T.H. Chan公共卫生学院）
Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions
Knight Foundation School of Computing and Information Sciences, Florida International University（佛罗里达国际大学奈特基金会计算与信息科学学院）；Security, Optimization, and Learning for InterDependent networks laboratory（相互依赖网络的安全、优化与学习实验室）；School of Computing, Southern Illinois University（南伊利诺伊大学计算机学院）；SambaNova Systems（SambaNova系统公司）
FlexLoc: Conditional Neural Networks for Zero-Shot Sensor Perspective Invariance in Object Localization with Distributed Multimodal Sensors
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；US DEVCOM Army Research Laboratory（美国DEVCOM陆军研究实验室）；Amazon（亚马逊）
BRIGHT: A globally distributed multimodal building damage assessment dataset with very-high-resolution for all-weather disaster response
Graduate School of Frontier Sciences, The University of Tokyo（东京大学前沿科学研究生院）；RIKEN Center for Advanced Intelligence Project (AIP), RIKEN（日本理化学研究院先进智能项目中心）；Department of Photogrammetry and Remote Sensing, ETH Zürich（苏黎世联邦理工学院测绘与遥感系）；Microsoft Research Asia（微软亚洲研究院）
Multimodal datasets contain an enormous amount of relational information, which grows exponentially with the introduction of new modalities. Learning representations in such a scenario is inherently complex due to the presence of multiple heterogeneous information channels. These channels can encode both (a) inter-relations between the items of different modalities and (b) intra-relations between the items of the same modality. Encoding multimedia items into a continuous low-dimensional semantic space such that both types of relations are captured and preserved is extremely challenging, especi…
University of Toronto（多伦多大学）；Canadian Institute for Advanced Research（加拿大高等研究院）
Inspired by recent advances in multimodal learning and machine translation, we introduce an encoder-decoder pipeline that learns (a): a multimodal joint embedding space with images and text and (b): a novel language model for decoding distributed representations from our space. Our pipeline effectively unifies joint image-text embedding models with multimodal neural language models. We introduce the structure-content neural language model that disentangles the structure of a sentence to its content, conditioned on representations produced by the encoder. The encoder allows one to rank images a…
CrossVIT-augmented Geospatial-Intelligence Visualization System for Tracking Economic Development Dynamics
Center for Applied Statistics（应用统计中心）；School of Statistics（统计学院）；Renmin University of China（中国人民大学）；School of Information（信息学院）；Department of Statistics and Applied Probability（统计与应用概率系）；University of California Santa Barbara（加州大学圣巴巴拉分校）
Spotlight on Token Perception for Multimodal Reinforcement Learning
Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；The Chinese University of Hong Kong（香港中文大学）；Nanjing University（南京大学）；Wuhan University（武汉大学）
