CIC-BART-SSA：基于结构化语义增强的可控图像描述生成
CIC-BART-SSA: Controllable Image Captioning with Structured Semantic Augmentation
AI总结：
提出CIC-BART-SSA模型，利用AMR结构化语义增强扩充数据集，生成多样且可控的图像描述，在多样性和文本质量上优于现有方法，并有效应对高度聚焦场景。
摘要：
可控图像描述生成（CIC）旨在根据终端用户提供的信息（例如感兴趣的区域、实体或事件）为图像生成自然语言描述。然而，现有的图像-语言数据集主要包含描述图像整体的标题，这使得它们难以有效训练能够关注任意区域或关系子集的CIC模型。为应对这一挑战，我们提出了一种新颖的全自动方法，利用基于图像现有标题集合构建的统一结构化语义表示，来采样额外的聚焦且视觉上可验证的描述。我们利用抽象语义表示（AMR）——一种跨语言的基于图的语义形式体系，来编码实体之间所有可能的时空语义关系，超越了当前方法通常仅关注空间关系的局限。我们使用这种结构化语义增强（SSA）框架，用可验证的可控描述来扩充现有的图像-标题数据集，增加其空间和语义多样性以及焦点覆盖范围。随后，我们开发了一个专门针对CIC任务的新模型CIC-BART-SSA，其控制信号来源于SSA多样化的数据集。我们通过实验表明，与最先进的CIC模型相比，CIC-BART-SSA生成的描述在多样性和文本质量方面更优，在可控性方面具有竞争力，并且重要的是，通过有效泛化到具有挑战性的高度聚焦场景，缩小了广泛与高度聚焦可控描述性能之间的差距。代码可在 https://github.com/SamsungLabs/CIC-BART-SSA 获取。
英文摘要：
Controllable Image Captioning (CIC) aims at generating natural language descriptions for an image, conditioned on information provided by end users, e.g., regions, entities or events of interest. However, available image-language datasets mainly contain captions that describe the entirety of an image, making them ineffective for training CIC models that can potentially attend to any subset of regions or relationships. To tackle this challenge, we propose a novel, fully automatic method to sample additional focused and visually grounded captions using a unified structured semantic representation built on top of the existing set of captions associated with an image. We leverage Abstract Meaning Representation (AMR), a cross-lingual graph-based semantic formalism, to encode all possible spatio-semantic relations between entities, beyond the typical spatial-relations-only focus of current methods. We use this Structured Semantic Augmentation (SSA) framework to augment existing image-caption datasets with the grounded controlled captions, increasing their spatial and semantic diversity and focal coverage. We then develop a new model, CIC-BART-SSA, specifically tailored for the CIC task, that sources its control signals from SSA-diversified datasets. We empirically show that, compared to SOTA CIC models, CIC-BART-SSA generates captions that are superior in diversity and text quality, are competitive in controllability, and, importantly, minimize the gap between broad and highly focused controlled captioning performance by efficiently generalizing to the challenging highly focused scenarios. Code is available at https://github.com/SamsungLabs/CIC-BART-SSA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
