arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.14804cs.CVcs.AIcs.CL
收藏

大型语言模型也能分享图像!

Large Language Models can Share Images, Too!

School of Computing, KAIST(韩国科学技术院计算机学院);KT Corporation(KT集团)

Young-Jun Lee, Dokyong Lee, Joo Won Sung, Jonghwan Hyeon, Ho-Jin Choi

AI总结:

本文提出DribeR框架并引入PhotoChat++数据集,首次在零样本设置下评估GPT-4、LLaMA 2等LLMs的图像分享能力,验证了其涌现能力及在人机交互和数据集增强中的实用性。

摘要:

本文探讨了大型语言模型(LLMs),如GPT-4和LLaMA 2,在零样本设置下的图像分享能力。为了便于对LLMs进行全面评估,我们引入了PhotoChat++数据集,该数据集包含丰富的标注信息(即意图、触发句、图像描述和显著信息)。此外,我们提出了无梯度且可扩展的Decide, Describe, and Retrieve(DribeR)框架。通过大量实验,我们解锁了配备LLMs的DribeR在零样本提示下的图像分享能力,其中ChatGPT取得了最佳性能。我们的发现还揭示了LLMs在零样本条件下涌现出的图像分享能力,验证了DribeR的有效性。我们使用该框架在两个真实场景中展示了其实用性和有效性:(1)人机交互和(2)数据集增强。据我们所知,这是第一项在零样本设置下评估各种LLMs图像分享能力的研究。我们在https://github.com/passing2961/DribeR上公开了源代码和数据集。

英文摘要:

This paper explores the image-sharing capability of Large Language Models (LLMs), such as GPT-4 and LLaMA 2, in a zero-shot setting. To facilitate a comprehensive evaluation of LLMs, we introduce the PhotoChat++ dataset, which includes enriched annotations (i.e., intent, triggering sentence, image description, and salient information). Furthermore, we present the gradient-free and extensible Decide, Describe, and Retrieve (DribeR) framework. With extensive experiments, we unlock the image-sharing capability of DribeR equipped with LLMs in zero-shot prompting, with ChatGPT achieving the best performance. Our findings also reveal the emergent image-sharing ability in LLMs under zero-shot conditions, validating the effectiveness of DribeR. We use this framework to demonstrate its practicality and effectiveness in two real-world scenarios: (1) human-bot interaction and (2) dataset augmentation. To the best of our knowledge, this is the first study to assess the image-sharing ability of various LLMs in a zero-shot setting. We make our source code and dataset publicly available at https://github.com/passing2961/DribeR.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑