arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.16355cs.CLcs.CV
收藏

PandaGPT:一个模型即可指令跟随所有模态

PandaGPT: One Model To Instruction-Follow Them All

University of Cambridge(剑桥大学);Nara Institute of Science and Technology(奈良先端科学技术大学院大学);Tencent AI Lab(腾讯人工智能实验室)

Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, Deng Cai

AI总结:

PandaGPT结合ImageBind多模态编码器与Vicuna大语言模型,仅用对齐图像-文本对训练,实现图像、视频、音频等多种模态的零样本指令跟随与语义组合。

摘要:

我们提出PandaGPT,一种赋予大型语言模型视觉与听觉指令跟随能力的方法。我们的初步实验表明,PandaGPT能够执行复杂任务,例如生成详细的图像描述、根据视频创作故事以及回答关于音频的问题。更有趣的是,PandaGPT可以同时接收多模态输入并自然地组合它们的语义。例如,PandaGPT能够将图像/视频中物体的外观与音频中物体的声音联系起来。为实现这一目标,PandaGPT结合了来自ImageBind的多模态编码器和来自Vicuna的大型语言模型。值得注意的是,训练PandaGPT仅需要对齐的图像-文本对。得益于ImageBind将不同模态的数据嵌入到同一空间中的强大能力,PandaGPT在图像和文本之外的数据(例如视频、音频、深度、热成像和IMU)上展现出涌现的,即零样本的跨模态行为。我们希望PandaGPT能够作为迈向通用人工智能的第一步,使其能够像人类一样整体地感知和理解不同模态的输入。我们的项目页面位于https://panda-gpt.github.io/。

英文摘要:

We present PandaGPT, an approach to emPower large lANguage moDels with visual and Auditory instruction-following capabilities. Our pilot experiments show that PandaGPT can perform complex tasks such as detailed image description generation, writing stories inspired by videos, and answering questions about audios. More interestingly, PandaGPT can take multimodal inputs simultaneously and compose their semantics naturally. For example, PandaGPT can connect how objects look in an image/video and how they sound in an audio. To do so, PandaGPT combines the multimodal encoders from ImageBind and the large language models from Vicuna. Notably, only aligned image-text pairs are required for the training of PandaGPT. Thanks to the strong capability of ImageBind in embedding data from different modalities into the same space, PandaGPT displays emergent, i.e. zero-shot, cross-modal behaviors for data other than image and text (e.g., video, audio, depth, thermal, and IMU). We hope that PandaGPT serves as an initial step toward building AGI that can perceive and understand inputs in different modalities holistically, as we humans do. Our project page is at https://panda-gpt.github.io/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑