CaptionSmiths：灵活控制图像描述生成中的语言模式
CaptionSmiths: Flexibly Controlling Language Pattern in Image Captioning
AI总结：
本文提出 CaptionSmiths 方法，通过将描述的长度、描述性和词语独特性量化为连续标量并利用端点向量插值进行条件控制，使单一图像描述模型能够平滑切换语言模式，并在词汇对齐和控制精度上优于基线。
摘要：
一个能够灵活切换其语言模式（例如描述性和长度）的图像描述生成模型应当具有广泛的用途，因为它可以应用于多种不同的应用场景。然而，尽管生成式视觉-语言模型取得了显著进步，对生成描述属性的细粒度控制仍然困难，原因有二：（i）现有模型在训练过程中并未将描述属性作为条件输入；（ii）现有模型无法平滑地将其语言模式从一种状态过渡到另一种状态。针对这一挑战，我们提出了一种新方法 CaptionSmiths，以获得一个能够处理多种语言模式的单一图像描述生成模型。首先，我们的方法在无需人工标注的情况下，将每条描述的三个属性——长度、描述性和词语独特性——量化为连续标量值。基于这些值，我们通过两个端点向量之间的插值来表示条件信息，这两个端点向量分别对应极端状态，例如一个用于非常短的描述，另一个用于非常长的描述。实验结果表明，所得模型能够平滑地改变输出描述的属性，并且相比基线方法表现出更高的词汇对齐度。例如，CaptionSmiths 在保持更好词汇对齐的同时，将控制描述长度的误差降低了506%。代码将发布在 https://github.com/omron-sinicx/captionsmiths。
英文摘要：
An image captioning model flexibly switching its language pattern, e.g., descriptiveness and length, should be useful since it can be applied to diverse applications. However, despite the dramatic improvement in generative vision-language models, fine-grained control over the properties of generated captions is not easy due to two reasons: (i) existing models are not given the properties as a condition during training and (ii) existing models cannot smoothly transition its language pattern from one state to the other. Given this challenge, we propose a new approach, CaptionSmiths, to acquire a single captioning model that can handle diverse language patterns. First, our approach quantifies three properties of each caption, length, descriptiveness, and uniqueness of a word, as continuous scalar values, without human annotation. Given the values, we represent the conditioning via interpolation between two endpoint vectors corresponding to the extreme states, e.g., one for a very short caption and one for a very long caption. Empirical results demonstrate that the resulting model can smoothly change the properties of the output captions and show higher lexical alignment than baselines. For instance, CaptionSmiths reduces the error in controlling caption length by 506\% despite better lexical alignment. Code will be available on https://github.com/omron-sinicx/captionsmiths.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
