arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.02547cs.CV
收藏

在多模态学习中利用视觉词元扩展文本上下文

Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learning

National University of Singapore(新加坡国立大学);Microsoft Gen AI(微软生成式人工智能部门);Central South University(中南大学)

Alex Jinpeng Wang, Linjie Li, Yiqi Lin, Min Li, Lijuan Wang, Mike Zheng Shou

AI总结:

本研究提出VisInContext方法,通过视觉词元处理长上下文文本,在几乎不增加计算成本下将56亿参数MOE模型的上下文长度从256扩展至2048,显著提升少样本评估及文档理解性能。

摘要:

由于大量的GPU内存和计算成本,使用更长的上下文长度训练模型是多模态模型面临的一项重大挑战。这项探索性研究并未提出最先进的模型;相反,它引入了一种创新方法,旨在高效地增加多模态大语言模型(MLLMs)中的上下文文本长度。我们提出了Visualized In-Context Text Processing(VisInContext,可视化上下文文本处理),该技术使用视觉词元来处理长上下文文本。该技术显著减少了训练和推理阶段的GPU内存使用和浮点运算次数(FLOPs)。例如,对于一个56亿参数的MOE模型,我们的方法在几乎相同的FLOPs下,将预训练的上下文文本长度从256个词元扩展到2048个词元。实验结果表明,使用VisInContext训练的模型在用于上下文少样本评估的常见下游基准测试中表现出色。此外,VisInContext与现有的增加上下文文本长度的方法互补,并增强了文档理解能力,在文档问答任务和顺序文档检索中展现出巨大潜力。

英文摘要:

Training models with longer in-context lengths is a significant challenge for multimodal model due to substantial GPU memory and computational costs. This exploratory study does not present state-of-the-art models; rather, it introduces an innovative method designed to increase in-context text length in multi-modality large language models (MLLMs) efficiently. We present Visualized In-Context Text Processing (VisInContext), which processes long in-context text using visual tokens. This technique significantly reduces GPU memory usage and floating point operations (FLOPs) for both training and inferenceing stage. For instance, our method expands the pre-training in-context text length from 256 to 2048 tokens with nearly same FLOPs for a 56 billion parameter MOE model. Experimental results demonstrate that model trained with VisInContext delivers superior performance on common downstream benchmarks for in-context few-shot evaluation. Additionally, VisInContext is complementary to existing methods for increasing in-context text length and enhances document understanding capabilities, showing great potential in document QA tasks and sequential document retrieval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑