arXiv 2412.200622025-01-15cs.CV

MADiff：基于掩码预测与注意力增强扩散的文本引导时尚图像编辑

MADiff: Text-Guided Fashion Image Editing with Mask Prediction and Attention-Enhanced Diffusion

Southeast University（东南大学）；Advanced Ocean Institute of Southeast University（东南大学先进海洋研究院）；Northwestern Polytechnical University（西北工业大学）；Binjiang Institute of Artificial Intelligence, ZJUT（浙江工业大学滨江人工智能研究院）；Alibaba Group（阿里巴巴集团）

Zechao Zhan, Dehong Gao, Jinxia Zhang, Jiale Huang, Yang Hu, Xin Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对时尚领域文本引导图像编辑存在的区域定位不准、编辑强度弱的问题，提出融合MaskNet与注意力增强扩散的MADiff模型，构建Fashion-E数据集，实验证明其性能优于现有最优方法。

摘要：

文本引导的图像编辑模型已在通用领域取得巨大成功。然而，将这些模型直接应用于时尚领域可能会遇到两个问题：（1）编辑区域定位不准确；（2）编辑强度不足。为解决这些问题，本文提出了MADiff模型。具体而言，为更精准地识别编辑区域，本文提出了MaskNet（掩码网络），该网络将来自大语言模型的前景区域、密集姿态（densepose）及掩码提示输入轻量级UNet，以预测编辑区域的掩码。为增强编辑强度，本文提出了注意力增强扩散模型，该模型将噪声图、注意力图以及MaskNet输出的掩码输入所提出的注意力处理器（Attention Processor），生成优化后的噪声图。通过将优化后的噪声图整合到扩散模型中，编辑后的图像能更好地与目标提示对齐。鉴于时尚图像编辑领域缺乏基准数据集，本文构建了名为Fashion-E的数据集，其训练集包含28390个图像-文本对，评估集包含2639个图像-文本对，覆盖四类时尚任务。在Fashion-E上的大量实验表明，与现有最优方法相比，本文所提方法能准确预测编辑区域的掩码，并显著提升时尚图像编辑的编辑强度。

英文摘要：

Text-guided image editing model has achieved great success in general domain. However, directly applying these models to the fashion domain may encounter two issues: (1) Inaccurate localization of editing region; (2) Weak editing magnitude. To address these issues, the MADiff model is proposed. Specifically, to more accurately identify editing region, the MaskNet is proposed, in which the foreground region, densepose and mask prompts from large language model are fed into a lightweight UNet to predict the mask for editing region. To strengthen the editing magnitude, the Attention-Enhanced Diffusion Model is proposed, where the noise map, attention map, and the mask from MaskNet are fed into the proposed Attention Processor to produce a refined noise map. By integrating the refined noise map into the diffusion model, the edited image can better align with the target prompt. Given the absence of benchmarks in fashion image editing, we constructed a dataset named Fashion-E, comprising 28390 image-text pairs in the training set, and 2639 image-text pairs for four types of fashion tasks in the evaluation set. Extensive experiments on Fashion-E demonstrate that our proposed method can accurately predict the mask of editing region and significantly enhance editing magnitude in fashion image editing compared to the state-of-the-art methods.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。