arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.03149cs.CV
收藏

CaMML:面向大模型的上下文感知多模态学习器

CaMML: Context-Aware Multimodal Learner for Large Models

The Chinese University of Hong Kong(香港中文大学);Amazon Web Services(亚马逊网络服务);The University of Chicago(芝加哥大学)

Yixin Chen, Shuai Zhang, Boran Han, Tong He, Bo Li

AI总结:

本研究提出轻量级上下文感知多模态学习器CaMML,通过分层设计高效整合多模态上下文样本,构建的CaMML-13B在十余个基准上超越LLaVA-1.5(13B),达到SOTA性能。

摘要:

在本研究中,我们提出了上下文感知多模态学习器(Context-Aware MultiModal Learner,CaMML),用于微调大型多模态模型(LMMs)。CaMML是一个轻量级模块,旨在将多模态上下文样本无缝整合到大模型中,从而使模型能够从相似的、特定领域的最新信息中获取知识,并做出有依据的推理。值得注意的是,CaMML具备高度可扩展性,凭借其分层设计能够高效处理冗长的多模态上下文示例。基于CaMML,我们开发了两个多模态模型CaMML-7B和CaMML-13B,它们在一系列多模态任务基准数据集上展现出卓越性能。尤为突出的是,CaMML-13B在十余个广受认可的多模态基准数据集上取得了当前最优性能,在未整合任何外部资源的情况下,以显著幅度超越了LLaVA-1.5(13B)。此外,我们开展了大量消融研究以探究CaMML的内部工作机制,并进行了定性分析以展示其处理现实世界挑战性案例的有效性。代码和模型可在以下链接获取:https://github.com/amazon-science/camml。

英文摘要:

In this work, we introduce Context-Aware MultiModal Learner (CaMML), for tuning large multimodal models (LMMs). CaMML, a lightweight module, is crafted to seamlessly integrate multimodal contextual samples into large models, thereby empowering the model to derive knowledge from analogous, domain-specific, up-to-date information and make grounded inferences. Importantly, CaMML is highly scalable and can efficiently handle lengthy multimodal context examples owing to its hierarchical design. Based on CaMML, we have developed two multimodal models, CaMML-7B and CaMML-13B, that have shown exceptional performance across an array of benchmark datasets for multimodal tasks. Remarkably, CaMML-13B achieves the state-of-the-art performance on over ten widely recognized multimodal benchmark datasets, surpassing LLaVA-1.5 (13B) with a noticeable margin, without integration of any external resources. Moreover, we have conducted extensive ablative studies to inspect the inner workings of CaMML and performed qualitative analyses to showcase its effectiveness in handling real-world challenging cases. Code and models are available at: https://github.com/amazon-science/camml.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑