arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.12226cs.CLcs.AIcs.CVcs.LG
收藏

AnyGPT:具有离散序列建模的统一多模态大语言模型

AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling

Fudan University(复旦大学);Multimodal Art Projection Research Community(多模态艺术投影研究社区);Shanghai AI Laboratory(上海人工智能实验室)
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, … 展开作者

Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yu-Gang Jiang, Xipeng Qiu

AI总结:

AnyGPT利用离散表示统一处理语音、文本、图像和音乐等多模态,无需改动LLM架构,通过数据级预处理和108k多轮对话指令数据,实现任意到任意的多模态对话,性能媲美专门模型。

摘要:

我们引入了AnyGPT,一种任意到任意的多模态语言模型,它利用离散表示来统一处理各种模态,包括语音、文本、图像和音乐。AnyGPT可以在不改变当前大语言模型(LLM)架构或训练范式的情况下稳定训练。相反,它仅依赖于数据层面的预处理,从而促进新模态无缝集成到LLM中,类似于新语言的融入。我们构建了一个以文本为中心的多模态数据集,用于多模态对齐预训练。利用生成模型,我们合成了第一个大规模任意到任意的多模态指令数据集。该数据集包含108k个多轮对话样本,这些对话错综复杂地交织了各种模态,从而使模型能够处理任意组合的多模态输入和输出。实验结果表明,AnyGPT能够促进任意到任意的多模态对话,同时在所有模态上实现与专门模型相当的性能,证明了离散表示可以有效地、便捷地在语言模型中统一多种模态。演示见https://junzhan2000.github.io/AnyGPT.github.io/。

英文摘要:

We introduce AnyGPT, an any-to-any multimodal language model that utilizes discrete representations for the unified processing of various modalities, including speech, text, images, and music. AnyGPT can be trained stably without any alterations to the current large language model (LLM) architecture or training paradigms. Instead, it relies exclusively on data-level preprocessing, facilitating the seamless integration of new modalities into LLMs, akin to the incorporation of new languages. We build a multimodal text-centric dataset for multimodal alignment pre-training. Utilizing generative models, we synthesize the first large-scale any-to-any multimodal instruction dataset. It consists of 108k samples of multi-turn conversations that intricately interweave various modalities, thus equipping the model to handle arbitrary combinations of multimodal inputs and outputs. Experimental results demonstrate that AnyGPT is capable of facilitating any-to-any multimodal conversation while achieving performance comparable to specialized models across all modalities, proving that discrete representations can effectively and conveniently unify multiple modalities within a language model. Demos are shown in https://junzhan2000.github.io/AnyGPT.github.io/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑