基于思维链子空间元学习的小样本图像描述（结合大型视觉与语言模型）
A Chain-of-Thought Subspace Meta-Learning for Few-shot Image Captioning with Large Vision and Language Models
AI总结：
针对小样本图像描述中视觉与语言模态领域差距大的问题，本文提出思维链子空间元学习框架，通过可调提示连接两个冻结大模型，在不同子空间学习各CoT步骤元参数，实验表明其性能优于基线。
摘要：
在海量数据上预训练的大规模视觉与语言模型编码了视觉和语言先验，这使其更容易生成更自然、更逼真的图像和语言。尽管如此，视觉和语言模态之间仍存在显著的领域差距，尤其是在小样本设置下训练数据稀缺时，此时可用于训练的数据非常有限。为缓解这一问题，本文提出一种多模态元学习框架，通过引入连接两个冻结预训练大型视觉与语言模型的可调提示，来弥合两者之间的差距。对于小样本图像描述，现有多模型元学习框架利用单步提示方案积累输入图像的视觉特征以引导语言模型，但仅用少量训练样本难以生成准确的图像描述。相反，我们提出思维链（CoT）元学习方案作为多步图像描述过程，以更好地模仿人类描述图像的方式。此外，我们进一步提出在不同子空间中学习对应每个CoT步骤的模型元参数，以避免干扰。我们在小样本设置下对三个常用图像描述数据集（MSCOCO、Flickr8k和Flickr30k）评估了所提方法。实验结果表明，我们的思维链子空间元学习策略在不同数据集上的不同指标下性能均优于基线方法。
英文摘要：
A large-scale vision and language model that has been pretrained on massive data encodes visual and linguistic prior, which makes it easier to generate images and language that are more natural and realistic. Despite this, there is still a significant domain gap between the modalities of vision and language, especially when training data is scarce in few-shot settings, where only very limited data are available for training. In order to mitigate this issue, a multi-modal meta-learning framework has been proposed to bridge the gap between two frozen pretrained large vision and language models by introducing a tunable prompt connecting these two large models. For few-shot image captioning, the existing multi-model meta-learning framework utilizes a one-step prompting scheme to accumulate the visual features of input images to guide the language model, which struggles to generate accurate image descriptions with only a few training samples. Instead, we propose a chain-of-thought (CoT) meta-learning scheme as a multi-step image captioning procedure to better imitate how humans describe images. In addition, we further propose to learn different meta-parameters of the model corresponding to each CoT step in distinct subspaces to avoid interference. We evaluated our method on three commonly used image captioning datasets, i.e., MSCOCO, Flickr8k, and Flickr30k, under few-shot settings. The results of our experiments indicate that our chain-of-thought subspace meta-learning strategy is superior to the baselines in terms of performance across different datasets measured by different metrics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
