MoAI:大型语言与视觉模型的全部智能混合
MoAI: Mixture of All Intelligence for Large Language and Vision Models
AI总结:
MoAI通过融合外部CV模型输出的辅助视觉信息与视觉、语言特征,利用专家混合机制,在零样本视觉语言任务中显著提升真实场景理解能力,无需扩大模型或额外数据。
摘要:
大型语言模型(LLMs)和指令微调的兴起,导致了当前指令微调大型语言与视觉模型(LLVMs)的趋势。这一趋势涉及要么精心策划大量针对特定目标的指令微调数据集,要么扩大LLVMs以管理海量的视觉语言(VL)数据。然而,当前的LLVMs忽视了在视觉感知任务(如分割、检测、场景图生成(SGG)和光学字符识别(OCR))中,从专门的计算机视觉(CV)模型可获得的详细且全面的真实世界场景理解。相反,现有的LLVMs主要依赖其LLM主干的大容量和涌现能力。因此,我们提出了一种新的LLVM,即全部智能混合(MoAI),它利用从外部分割、检测、SGG和OCR模型的输出中获得的辅助视觉信息。MoAI通过两个新引入的模块运行:MoAI-Compressor和MoAI-Mixer。在将外部CV模型的输出语言化后,MoAI-Compressor对齐并压缩它们,以高效地利用相关的辅助视觉信息用于VL任务。然后,MoAI-Mixer利用专家混合的概念,融合三种类型的智能:(1)视觉特征,(2)来自外部CV模型的辅助特征,以及(3)语言特征。通过这种整合,MoAI在众多零样本VL任务中显著优于开源和闭源LLVMs,特别是在与真实世界场景理解相关的任务中,如物体存在性、位置、关系和OCR,而无需扩大模型规模或策划额外的视觉指令微调数据集。
英文摘要:
The rise of large language models (LLMs) and instruction tuning has led to the current trend of instruction-tuned large language and vision models (LLVMs). This trend involves either meticulously curating numerous instruction tuning datasets tailored to specific objectives or enlarging LLVMs to manage vast amounts of vision language (VL) data. However, current LLVMs have disregarded the detailed and comprehensive real-world scene understanding available from specialized computer vision (CV) models in visual perception tasks such as segmentation, detection, scene graph generation (SGG), and optical character recognition (OCR). Instead, the existing LLVMs rely mainly on the large capacity and emergent capabilities of their LLM backbones. Therefore, we present a new LLVM, Mixture of All Intelligence (MoAI), which leverages auxiliary visual information obtained from the outputs of external segmentation, detection, SGG, and OCR models. MoAI operates through two newly introduced modules: MoAI-Compressor and MoAI-Mixer. After verbalizing the outputs of the external CV models, the MoAI-Compressor aligns and condenses them to efficiently use relevant auxiliary visual information for VL tasks. MoAI-Mixer then blends three types of intelligence (1) visual features, (2) auxiliary features from the external CV models, and (3) language features by utilizing the concept of Mixture of Experts. Through this integration, MoAI significantly outperforms both open-source and closed-source LLVMs in numerous zero-shot VL tasks, particularly those related to real-world scene understanding such as object existence, positions, relations, and OCR without enlarging the model size or curating extra visual instruction tuning datasets.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。