Caption Anything：具有多样化多模态控制的交互式图像描述
Caption Anything: Interactive Image Description with Diverse Multimodal Controls
AI总结：
本文提出Caption AnyThing（CAT）框架，结合SAM和ChatGPT，统一视觉与语言控制，支持点、框、轨迹及情感、长度等多种多模态控制，实现灵活交互式图像描述。
摘要：
可控图像描述是一个新兴的多模态课题，旨在按照人类目的用自然语言描述图像，例如查看指定区域或以特定文本风格进行叙述。最先进的方法是在输入控制与输出描述的标注对上进行训练。然而，此类标注良好的多模态数据的稀缺性在很大程度上限制了它们在交互式AI系统中的可用性和可扩展性。利用单模态指令跟随基础模型是一种有前景的替代方案，可以受益于更广泛的数据来源。在本文中，我们提出了Caption AnyThing（CAT），这是一个由基础模型增强的图像描述框架，支持广泛的多模态控制：1）视觉控制，包括点、框和轨迹；2）语言控制，如情感、长度、语言和事实性。借助Segment Anything Model（SAM）和ChatGPT，我们将视觉和语言提示统一到一个模块化框架中，实现不同控制之间的灵活组合。大量案例研究证明了我们框架的用户意图对齐能力，为视觉-语言应用中有效的用户交互建模提供了启示。我们的代码已在https://github.com/ttengwang/Caption-Anything公开提供。
英文摘要：
Controllable image captioning is an emerging multimodal topic that aims to describe the image with natural language following human purpose, $\textit{e.g.}$, looking at the specified regions or telling in a particular text style. State-of-the-art methods are trained on annotated pairs of input controls and output captions. However, the scarcity of such well-annotated multimodal data largely limits their usability and scalability for interactive AI systems. Leveraging unimodal instruction-following foundation models is a promising alternative that benefits from broader sources of data. In this paper, we present Caption AnyThing (CAT), a foundation model augmented image captioning framework supporting a wide range of multimodel controls: 1) visual controls, including points, boxes, and trajectories; 2) language controls, such as sentiment, length, language, and factuality. Powered by Segment Anything Model (SAM) and ChatGPT, we unify the visual and language prompts into a modularized framework, enabling the flexible combination between different controls. Extensive case studies demonstrate the user intention alignment capabilities of our framework, shedding light on effective user interaction modeling in vision-language applications. Our code is publicly available at https://github.com/ttengwang/Caption-Anything.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
