实例感知图像着色：可控文本描述与分割掩码
Instance-aware Image Colorization with Controllable Textual Descriptions and Segmentation Masks
AI总结：
提出基于扩散的MT-Color方法，利用分割掩码和文本引导实现实例级图像着色，解决颜色渗漏与绑定错误，并构建GPT-color数据集，实验验证其优越性。
摘要：
近年来，深度学习在图像着色中的应用受到了广泛关注。扩散模型的成熟进一步推动了图像着色模型的发展。然而，当前主流的图像着色模型仍面临颜色渗漏和颜色绑定错误等问题，并且无法在实例级别对图像进行着色。在本文中，我们提出了一种基于扩散的着色方法MT-Color，以利用用户提供的指导实现精确的实例感知着色。为了解决颜色渗漏问题，我们设计了一种像素级掩码注意力机制，通过交叉注意力整合潜在特征和条件灰度图像特征。我们使用分割掩码构建交叉注意力掩码，防止不同实例之间的像素信息交换。我们还引入了一个实例掩码和文本引导模块，该模块提取每个实例的实例掩码和文本表示，然后通过自注意力将其与潜在特征融合，利用实例掩码形成自注意力掩码，以防止实例文本引导其他区域的着色，从而缓解颜色绑定错误。此外，我们采用了一种多实例采样策略，即分别对每个实例区域进行采样，然后融合结果。另外，我们通过利用现有图像数据集上的大型视觉语言模型，创建了一个专门用于实例级着色任务的数据集GPT-color。定性和定量实验表明，我们的模型和数据集优于以往的方法和数据集。
英文摘要：
Recently, the application of deep learning in image colorization has received widespread attention. The maturation of diffusion models has further advanced the development of image colorization models. However, current mainstream image colorization models still face issues such as color bleeding and color binding errors, and cannot colorize images at the instance level. In this paper, we propose a diffusion-based colorization method MT-Color to achieve precise instance-aware colorization with use-provided guidance. To tackle color bleeding issue, we design a pixel-level mask attention mechanism that integrates latent features and conditional gray image features through cross-attention. We use segmentation masks to construct cross-attention masks, preventing pixel information from exchanging between different instances. We also introduce an instance mask and text guidance module that extracts instance masks and text representations of each instance, which are then fused with latent features through self-attention, utilizing instance masks to form self-attention masks to prevent instance texts from guiding the colorization of other areas, thus mitigating color binding errors. Furthermore, we apply a multi-instance sampling strategy, which involves sampling each instance region separately and then fusing the results. Additionally, we have created a specialized dataset for instance-level colorization tasks, GPT-color, by leveraging large visual language models on existing image datasets. Qualitative and quantitative experiments show that our model and dataset outperform previous methods and datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
