可控情境化图像描述：通过用户定义的高亮引导视觉叙事
Controllable Contextualized Image Captioning: Directing the Visual Narrative through User-Defined Highlights
AI总结：
本文提出可控情境化图像描述（Ctrl-CIC），通过用户定义高亮引导描述生成，采用P-Ctrl和R-Ctrl两种方法，并设计GPT-4V评估器，实验证明其高效可控，为自适应图像描述开辟新方向。
摘要：
情境化图像描述（CIC）将传统图像描述发展为一个更复杂的领域，需要具备多模态推理能力。其目标是在给定特定情境信息的情况下生成图像描述。本文进一步引入了一个新领域：可控情境化图像描述（Ctrl-CIC）。与仅依赖广泛情境的CIC不同，Ctrl-CIC强调用户定义的高亮，迫使模型生成与情境中高亮方面相呼应的描述。我们提出了两种方法：基于提示的控制器（P-Ctrl）和基于重新校准的控制器（R-Ctrl），以生成聚焦的描述。P-Ctrl通过在高亮前添加高亮驱动的前缀来条件化模型生成，而R-Ctrl则调整模型以选择性地重新校准编码器对高亮标记的嵌入。此外，我们设计了一个由GPT-4V驱动的评估器，与标准评估方法一起评估受控描述的质量。大量实验结果表明，我们的方法具有高效且有效的可控性，为实现用户自适应的图像描述开辟了新方向。代码可在 https://github.com/ShunqiM/Ctrl-CIC 获取。
英文摘要：
Contextualized Image Captioning (CIC) evolves traditional image captioning into a more complex domain, necessitating the ability for multimodal reasoning. It aims to generate image captions given specific contextual information. This paper further introduces a novel domain of Controllable Contextualized Image Captioning (Ctrl-CIC). Unlike CIC, which solely relies on broad context, Ctrl-CIC accentuates a user-defined highlight, compelling the model to tailor captions that resonate with the highlighted aspects of the context. We present two approaches, Prompting-based Controller (P-Ctrl) and Recalibration-based Controller (R-Ctrl), to generate focused captions. P-Ctrl conditions the model generation on highlight by prepending captions with highlight-driven prefixes, whereas R-Ctrl tunes the model to selectively recalibrate the encoder embeddings for highlighted tokens. Additionally, we design a GPT-4V empowered evaluator to assess the quality of the controlled captions alongside standard assessment methods. Extensive experimental results demonstrate the efficient and effective controllability of our method, charting a new direction in achieving user-adaptive image captioning. Code is available at https://github.com/ShunqiM/Ctrl-CIC .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
