AlignGPT：具备自适应对齐能力的多模态大型语言模型
AlignGPT: Multi-modal Large Language Models with Adaptive Alignment Capability
AI总结：
针对现有多模态大型语言模型在跨模态对齐建模中忽视图文对对齐程度差异及任务差异化对齐需求的问题，提出AlignGPT，通过预训练分组合并不同对齐水平表示、指令微调自适应组合表示，在12个基准测试中取得竞争力性能。
摘要：
多模态大型语言模型（MLLMs）被广泛认为是探索人工通用智能（AGI）的关键。MLLMs的核心在于其实现跨模态对齐的能力。为实现这一目标，当前MLLMs通常遵循两阶段训练范式：预训练阶段和指令微调阶段。尽管取得了成功，这些模型在对齐能力建模方面仍存在不足。首先，在预训练阶段，模型通常假设所有图文对都是均匀对齐的，但实际上不同图文对的对齐程度并不一致。其次，当前用于微调的指令包含多种任务，不同任务通常需要不同水平的对齐能力，但以往的MLLMs忽视了这些差异化的对齐需求。为解决这些问题，我们提出了一种新的多模态大型语言模型AlignGPT。在预训练阶段，我们不再平等对待所有图文对，而是根据它们的对齐程度将其分为不同组，然后训练模型学习不同对齐水平的表示。在指令微调阶段，我们自适应地组合这些对齐水平的表示，以满足不同任务的动态对齐需求。大量实验结果表明，我们的模型在12个基准测试中取得了具有竞争力的性能。
英文摘要：
Multimodal Large Language Models (MLLMs) are widely regarded as crucial in the exploration of Artificial General Intelligence (AGI). The core of MLLMs lies in their capability to achieve cross-modal alignment. To attain this goal, current MLLMs typically follow a two-phase training paradigm: the pre-training phase and the instruction-tuning phase. Despite their success, there are shortcomings in the modeling of alignment capabilities within these models. Firstly, during the pre-training phase, the model usually assumes that all image-text pairs are uniformly aligned, but in fact the degree of alignment between different image-text pairs is inconsistent. Secondly, the instructions currently used for finetuning incorporate a variety of tasks and different tasks usually require different levels of alignment capabilities, but previous MLLMs overlook these differentiated alignment needs. To tackle these issues, we propose a new multimodal large language model AlignGPT. In the pre-training stage, instead of treating all image-text pairs equally, we divide them into different groups according to the degrees of alignment of them. Then, the model is trained to learn the representations of different alignment levels. In the instruction-tuning phase, we adaptively combine these representations of alignment levels to meet the dynamic alignment needs of different tasks. Extensive experimental results show that our model achieves competitive performance on 12 benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
