由于大量的GPU内存和计算成本,使用更长的上下文长度训练模型是多模态模型面临的一项重大挑战。这项探索性研究并未提出最先进的模型;相反,它引入了一种创新方法,旨在高效地增加多模态大语言模型(MLLMs)中的上下文文本长度。我们提出了Visualized In-Context Text Processing(VisInContext,可视化上下文文本处理),该技术使用视觉词元来处理长上下文文本。该技术显著减少了训练和推理阶段的GPU内存使用和浮点运算次数(FLOPs)。例如,对于一个56亿参数的MOE模型,我们的方法在几乎相同的FLOPs下,将预训练的上下文文本长度从256个词元扩展到2048个词元。实验结果表明,使用VisInContext训练的模型在用于上下文少样本评估的常见下游基准测试中表现出色。此外,VisInContext与现有的增加上下文文本长度的方法互补,并增强了文档理解能力,在文档问答任务和顺序文档检索中展现出巨大潜力。
英文摘要:
Training models with longer in-context lengths is a significant challenge for multimodal model due to substantial GPU memory and computational costs. This exploratory study does not present state-of-the-art models; rather, it introduces an innovative method designed to increase in-context text length in multi-modality large language models (MLLMs) efficiently. We present Visualized In-Context Text Processing (VisInContext), which processes long in-context text using visual tokens. This technique significantly reduces GPU memory usage and floating point operations (FLOPs) for both training and inferenceing stage. For instance, our method expands the pre-training in-context text length from 256 to 2048 tokens with nearly same FLOPs for a 56 billion parameter MOE model. Experimental results demonstrate that model trained with VisInContext delivers superior performance on common downstream benchmarks for in-context few-shot evaluation. Additionally, VisInContext is complementary to existing methods for increasing in-context text length and enhances document understanding capabilities, showing great potential in document QA tasks and sequential document retrieval.
Tsinghua University(清华大学);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Institute for AI Industry Research (AIR), Tsinghua University(清华大学智能产业研究院(AIR))
Sparsity Meets Similarity: Leveraging Long-Tail Distribution for Dynamic Optimized Token Representation in Multimodal Large Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institution of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Cross-Modal Action Recognition in Egocentric Video Using Mamba: Integrating RGB and Hand Skeleton Streams via CLS Token Fusion Strategies
Univ. of Buenos Aires. Faculty of Exact and Natural Sciences. Dept. of Computer Science (DC)(布宜诺斯艾利斯大学。精确与自然科学学院。计算机科学系);CONICET-Univ. of Buenos Aires. Institute of Computer Sciences (ICC)(布宜诺斯艾利斯大学CONICET联合体。计算机科学研究所)
Efficient Multi-modal Large Language Models via Progressive Consistency Distillation
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Duke University(杜克大学);The University of Hong Kong(香港大学);Peking University(北京大学);University of Chicago(芝加哥大学);Sun Yat-sen University(中山大学)
Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification
School of Future Technology(未来技术学院);School of Artificial Intelligence(人工智能学院);Dalian University of Technology(大连理工大学);Anhui University(安徽大学);School of Computer Science and Technology(计算机科学与技术学院)
OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉科学学院);Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所中文信息处理实验室);University of Limerick(利默里克大学);CUHK, Shenzhen(香港中文大学(深圳))
Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models
CVIU Lab, University of Arkansas, USA(美国亚拉巴马大学CVIU实验室);Vietnam National University, Ho Chi Minh City University of Science, Vietnam(越南国家大学胡志明市科技大学);Carnegie Mellon University, USA(美国卡内基梅隆大学)
SDPT: Synchronous Dual Prompt Tuning for Fusion-based Visual-Language Pre-trained Models
School of Biological Science and Medical Engineering, Beihang University(北京航空航天大学生物与医学工程学院);ByteDance Inc.(字节跳动公司);Zhejiang University(浙江大学);Taiwan Artificial Intelligence Foundation(台湾人工智能基金会)
2D or 3D: Who Governs Salience in VLA Models? -- Tri-Stage Token Pruning Framework with Modality Salience Awareness
School of Computer Science, Peking University(北京大学计算机科学学院);ZTE Corporation(中兴通讯);School of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院);School of Computer Science, China University of Geosciences (Wuhan)(中国地质大学(武汉)计算机科学学院);School of Electronics Engineering and Computer Science, Peking University(北京大学电子工程与计算机科学学院)