arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.01293cs.LGcs.CL
收藏

多模态大语言模型能否执行文本到图像的情境学习?

Can MLLMs Perform Text-to-Image In-Context Learning?

University of Wisconsin-Madison(威斯康星大学麦迪逊分校);FuriosaAI(富力沙AI公司);Seoul National University(首尔大学);Ajou University(亚洲大学)

Yuchen Zeng, Wonjun Kang, Yicong Chen, Hyung Il Koo, Kangwook Lee

AI总结:

本文正式定义文本到图像情境学习(T2I-ICL)任务,构建首个基准数据集CoBSAT,通过评测六个多模态大语言模型揭示其困难,并提出微调与思维链提示等缓解策略以显著提升性能。

摘要:

从大语言模型(LLMs)到多模态大语言模型(MLLMs)的演进,推动了将情境学习(ICL)扩展至其多模态对应形式的研究。现有此类研究主要集中在图像到文本的情境学习(image-to-text ICL)上。然而,文本到图像的情境学习(T2I-ICL)因其独特特性和潜在应用,仍未得到充分探索。为填补这一空白,我们正式定义了T2I-ICL任务,并提出了CoBSAT,这是首个T2I-ICL基准数据集,涵盖十个任务。利用我们的数据集对六个最先进的MLLMs进行基准测试,我们揭示了MLLMs在解决T2I-ICL时遇到的巨大困难。我们识别出主要挑战在于多模态和图像生成固有的复杂性,并表明微调和思维链(Chain-of-Thought)提示等策略有助于缓解这些困难,从而带来性能的显著提升。我们的代码和数据集可在https://github.com/UW-Madison-Lee-Lab/CoBSAT获取。

英文摘要:

The evolution from Large Language Models (LLMs) to Multimodal Large Language Models (MLLMs) has spurred research into extending In-Context Learning (ICL) to its multimodal counterpart. Existing such studies have primarily concentrated on image-to-text ICL. However, the Text-to-Image ICL (T2I-ICL), with its unique characteristics and potential applications, remains underexplored. To address this gap, we formally define the task of T2I-ICL and present CoBSAT, the first T2I-ICL benchmark dataset, encompassing ten tasks. Utilizing our dataset to benchmark six state-of-the-art MLLMs, we uncover considerable difficulties MLLMs encounter in solving T2I-ICL. We identify the primary challenges as the inherent complexity of multimodality and image generation, and show that strategies such as fine-tuning and Chain-of-Thought prompting help to mitigate these difficulties, leading to notable improvements in performance. Our code and dataset are available at https://github.com/UW-Madison-Lee-Lab/CoBSAT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑