扩展多模态大语言模型（MLLMs）的长上下文能力对于推进视频理解和高分辨率图像分析至关重要。实现这一目标需要在模型架构、数据构建和训练策略上进行系统性改进，特别是要应对图像数量增加时性能下降和高计算成本等挑战。在本文中，我们提出了一种融合Mamba与Transformer模块的混合架构，引入了能够同时捕捉时间依赖和空间依赖的数据构建方法，并采用了渐进式训练策略。我们发布的模型LongLLaVA（Long-Context Large Language and Vision Assistant，长上下文大语言与视觉助手）展示了效率与性能之间的有效平衡。LongLLaVA在多个基准测试上取得了具有竞争力的结果，同时保持了高吞吐量和低内存消耗。值得注意的是，它可以在单块A100 80GB GPU上处理近一千张图像，凸显了其在广泛多模态应用中的潜力。
英文摘要：
Expanding the long-context capabilities of Multi-modal Large Language Models~(MLLMs) is critical for advancing video understanding and high-resolution image analysis. Achieving this requires systematic improvements in model architecture, data construction, and training strategies, particularly to address challenges such as performance degradation with increasing image counts and high computational costs. In this paper, we propose a hybrid architecture that integrates Mamba and Transformer blocks, introduce data construction methods that capture both temporal and spatial dependencies, and employ a progressive training strategy. Our released model, LongLLaVA (\textbf{Long}-Context \textbf{L}arge \textbf{L}anguage \textbf{a}nd \textbf{V}ision \textbf{A}ssistant), demonstrates an effective balance between efficiency and performance. LongLLaVA achieves competitive results across various benchmarks while maintaining high throughput and low memory consumption. Notably, it can process nearly one thousand images on a single A100 80GB GPU, underscoring its potential for a wide range of multi-modal applications.
DualEdit: Dual Editing for Knowledge Updating in Vision-Language Models
Harvard University（哈佛大学）；City University of Hong Kong（香港城市大学）；Amazon（亚马逊）；Fudan University（复旦大学）；MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（人工智能关键实验室，上海交通大学）
IRONIC: Coherence-Aware Reasoning Chains for Multi-Modal Sarcasm Detection
College of Information Sciences and Technology（信息科学与技术学院）；The Pennsylvania State University（宾夕法尼亚州立大学）；Department of Computer Science and Engineering（计算机科学与工程系）；National Institute of Technology, Tiruchirappalli（特里奇里帕利理工学院）
Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameters
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机科学学院，北京大学）；WeChat AI, Tencent Inc., China（腾讯公司，中国）；The University of Hong Kong（香港大学）
CoLA: Cross-Modal Low-rank Adaptation for Multimodal Downstream Tasks
Centre for Vision, Speech and Signal Processing (CVSSP)（视觉、语音和信号处理中心）；University of Surrey（塞维利亚大学）；Surrey Institute for People-Centred AI（以人为本的人工智能研究所）
Efficient Long-distance Latent Relation-aware Graph Neural Network for Multi-modal Emotion Recognition in Conversations
Central South University of Forestry and Technology（中南林业科技大学）；Changsha Medical University（长沙医学院）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
PheME: A deep ensemble framework for improving phenotype prediction from multi-modal data
Xi’an Jiaotong-Liverpool University（西交利物浦大学）；Wuhan University（武汉大学）；Rice University（莱斯大学）；Texas A&M University（德克萨斯农工大学）；University of Texas Health Science Center（德克萨斯大学健康科学中心）；UTHealth Houston School of Biomedical Informatics（德克萨斯大学休斯顿健康科学中心生物医学信息学院）
本文提出 PheME 深度集成框架，融合结构化 EHR 与临床笔记的单模态和多模态模型，以提升七种疾病的表型预测性能。
Collaboration of Fusion and Independence: Hypercomplex-driven Robust Multi-Modal Knowledge Graph Completion
School of Software Technology, Zhejiang University（浙江大学软件学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；ZJU-Ant Group Joint Lab of Knowledge Graph（浙大蚂蚁集团知识图谱联合实验室）；Ant Group（蚂蚁集团）
