arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.07490cs.CLcs.CV
收藏

ArtGPT-4:迈向具有增强适配器的艺术理解大型视觉-语言模型

ArtGPT-4: Towards Artistic-understanding Large Vision-Language Models with Enhanced Adapter

University of Notre Dame(圣母大学);Anhui Polytechnic University(安徽工程大学);Lehigh University(理海大学)

Zhengqing Yuan, Yunhong He, Kun Wang, Yanfang Ye, Lichao Sun

AI总结:

本文提出 ArtGPT-4,通过在 LLM 中集成专用增强适配器高效理解艺术图像,仅用约 0.52M 图文对训练 2 小时,便在多个艺术情感数据集上达到最先进水平。

摘要:

大型语言模型(LLM)的成功激发了多模态学习这一新兴研究领域。然而,利用 LLM 开展多模态学习的一项重大挑战在于预训练 LLM 的规模,它们通常拥有数十亿参数。为应对这一挑战,MiniGPT-4 和 LLaVA 等模型已被开发出来,用更少参数对预训练模型进行微调。尽管这些模型表现出良好性能,但它们对艺术图像的理解仍然有限。为促进更好的艺术理解,本文提出 ArtGPT-4,这是一个开创性的大型视觉-语言模型,专门用于解决现有模型在艺术理解方面的局限。ArtGPT-4 的关键创新在于其针对艺术图像理解这一复杂挑战的精心设计,这使其区别于其他忽视细微细节而关注更宽泛主题的模型。具体而言,该模型通过将若干专用 adapter(适配器)层集成到 LLM 中发挥作用,使模型能够更高效、更有效地解析和解释复杂视觉 token,而不是像现有方法那样微调整个 LLM。ArtGPT-4 已展示出卓越的效率表现:使用一台 Tesla A100 设备,借助包含约 0.52M 条条目的图文对数据集,其训练仅需 2 小时即可完成。此外,ArtGPT-4 还在 ArtEmis 和 ArtEmis-v2.0 数据集以及本工作建立的基准上取得了最先进性能;在 6 分量表上,它与专业艺术家描述之间仅相差可忽略的 0.15 分。ArtGPT-4 的出色性能表明,它能够以艺术理解来呈现图像,并传达图像所激发的情感,从而反映人类的解读。代码和预训练模型可在 https://github.com/DLYuanGod/ArtGPT-4 获取。

英文摘要:

The success of large language models (LLMs) has inspired an emerging research field of multimodal learning. However, a grand challenge of exploiting LLMs for multimodal learning is the size of pre-trained LLMs which are always with billions of parameters. To tackle this challenge, models such as MiniGPT-4 and LLaVA have been developed to fine-tune the pre-trained models using fewer parameters. Despite their promising performance, these models remain limited in their understanding of artistic imagery. To facilitate better artistic-understanding, in this paper, we propose ArtGPT-4, a pioneering large vision-language model tailored to address the limitations of existing models in artistic comprehension. The key innovation of ArtGPT-4 lies in its craft for the sophisticated challenge of artistic image comprehension, setting it apart from other models that overlook fine details for broader themes. Specifically, it works by integrating some specialized adapter layers into the LLM, enabling the model to more efficiently and effectively parse and interpret complex visual tokens, instead of fine-tuning the whole LLM as in the existing method. ArtGPT-4 has demonstrated its outstanding performance on the efficiency: utilizing a Tesla A100 device, its training can be completed in mere 2 hours with an image-text pair dataset comprising approximately 0.52M entries. Additionally, ArtGPT-4 has also achieved state-of-the-art performance on the ArtEmis and ArtEmis-v2.0 datasets as well as the benchmarks established in this work, lagging behind professional artists' descriptions by a negligible 0.15 points on a 6-point scale. The outstanding performance of ArtGPT-4 shows that it can render images with an artistic-understanding and convey the emotions they inspire, mirroring human interpretation. The code and the pre-trained model are accessible in \url{https://github.com/DLYuanGod/ArtGPT-4}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑