学习用于通用可控图像描述的组合提示
Learning Combinatorial Prompts for Universal Controllable Image Captioning
AI总结：
提出基于组合提示的框架ComPro，利用GPT-2与轻量级提示生成网络及掩码注意力机制，统一实现单一与组合控制信号的可控图像描述。
摘要：
可控图像描述（Controllable Image Captioning, CIC）——在给定控制信号的指导下生成关于图像的自然语言描述——是迈向下一代描述系统最有前景的方向之一。迄今为止，已提出多种CIC控制信号，涵盖从内容相关控制到结构相关控制。然而，由于不同控制信号在格式和目标上的差异，所有现有CIC工作（或架构）仅聚焦于某一种控制信号，忽略了类人的组合能力。所谓“组合”，是指我们人类在生成描述时能轻松同时满足多种需求（或约束）。为此，我们提出一种基于提示的新型CIC框架，通过学习组合提示，称为ComPro。具体而言，我们直接利用预训练语言模型GPT-2作为语言模型，有助于弥合不同特定信号CIC架构间的差距。接着，我们将CIC重新表述为提示引导的句子生成问题，并提出一种新型轻量级提示生成网络，为不同控制信号生成组合提示。针对不同控制信号，我们进一步设计新型掩码注意力机制以实现基于提示的CIC。因其简洁性，ComPro可通过拼接这些提示进一步扩展至更多类型的组合控制信号。在两个主流CIC基准上的大量实验验证了ComPro在单一和组合控制信号上的有效性与效率。
英文摘要：
Controllable Image Captioning (CIC) -- generating natural language descriptions about images under the guidance of given control signals -- is one of the most promising directions towards next-generation captioning systems. Till now, various kinds of control signals for CIC have been proposed, ranging from content-related control to structure-related control. However, due to the format and target gaps of different control signals, all existing CIC works (or architectures) only focus on one certain control signal, and overlook the human-like combinatorial ability. By ``combinatorial", we mean that our humans can easily meet multiple needs (or constraints) simultaneously when generating descriptions. To this end, we propose a novel prompt-based framework for CIC by learning Combinatorial Prompts, dubbed as ComPro. Specifically, we directly utilize a pretrained language model GPT-2 as our language model, which can help to bridge the gap between different signal-specific CIC architectures. Then, we reformulate the CIC as a prompt-guide sentence generation problem, and propose a new lightweight prompt generation network to generate the combinatorial prompts for different kinds of control signals. For different control signals, we further design a new mask attention mechanism to realize the prompt-based CIC. Due to its simplicity, our ComPro can be further extended to more kinds of combined control signals by concatenating these prompts. Extensive experiments on two prevalent CIC benchmarks have verified the effectiveness and efficiency of our ComPro on both single and combined control signals.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
