arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.09093cs.CLcs.AIcs.CV
收藏

Macaw-LLM:集成图像、音频、视频与文本的多模态语言建模

Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration

Tencent AI Lab(腾讯人工智能实验室);Dublin City University(都柏林城市大学);Monash University(莫纳什大学)

Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, Zhaopeng Tu

AI总结:

针对指令微调大语言模型在非文本模态效能研究不足的问题,提出集成多模态的Macaw-LLM,含模态、认知、对齐三大模块,构建含69K图像、50K视频的多轮对话指令数据集并开源相关资源,助力多模态LLM研究。

摘要:

尽管经过指令微调的大语言模型(LLM)已在各类自然语言处理任务中展现出卓越能力,但其在文本之外其他数据模态上的效能尚未得到充分研究。本研究提出Macaw-LLM,这是一种可无缝整合视觉、音频与文本信息的新型多模态大语言模型。Macaw-LLM包含三大核心组件:用于编码多模态数据的模态模块、利用预训练LLM的认知模块,以及用于协调不同表征的对齐模块。我们提出的新型对齐模块可将多模态特征与文本特征无缝桥接,简化了从模态模块到认知模块的适配过程。此外,我们构建了一个基于多轮对话的大规模多模态指令数据集,包含6.9万个图像样本与5万个视频样本。我们已将数据、代码与模型公开,希望能为多模态大语言模型的后续研究铺路,拓展LLM处理多样数据模态、应对复杂现实场景的能力。

英文摘要:

Although instruction-tuned large language models (LLMs) have exhibited remarkable capabilities across various NLP tasks, their effectiveness on other data modalities beyond text has not been fully studied. In this work, we propose Macaw-LLM, a novel multi-modal LLM that seamlessly integrates visual, audio, and textual information. Macaw-LLM consists of three main components: a modality module for encoding multi-modal data, a cognitive module for harnessing pretrained LLMs, and an alignment module for harmonizing diverse representations. Our novel alignment module seamlessly bridges multi-modal features to textual features, simplifying the adaptation process from the modality modules to the cognitive module. In addition, we construct a large-scale multi-modal instruction dataset in terms of multi-turn dialogue, including 69K image instances and 50K video instances. We have made our data, code and model publicly available, which we hope can pave the way for future research in multi-modal LLMs and expand the capabilities of LLMs to handle diverse data modalities and address complex real-world scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑