PromptDresser：通过生成式文本提示和提示感知掩码提升虚拟试穿的质量与可控性
PromptDresser: Improving the Quality and Controllability of Virtual Try-On via Generative Textual Prompt and Prompt-aware Mask
AI总结：
提出PromptDresser，利用大型多模态模型生成文本提示并自适应调整掩码，解决文本可编辑虚拟试穿中的描述生成、信息冲突和编辑区域问题，显著提升生成质量与可控性。
摘要：
近期的虚拟试穿方法通过微调预训练的文本到图像扩散模型来利用其强大的生成能力，取得了进展。然而，文本提示在虚拟试穿中的应用仍未得到充分探索。本文处理一个文本可编辑的虚拟试穿任务，即根据提供的服装图像修改服装，同时根据文本描述编辑穿着风格（例如，塞入方式、合身度）。在文本可编辑的虚拟试穿中，存在三个关键方面：（i）为配对的个人-服装数据设计丰富的文本描述以训练模型，（ii）解决现有个人服装的文本信息干扰新服装生成时的冲突，（iii）自适应地调整与文本描述对齐的修复掩码，确保适当的编辑区域，同时保留与新服装无关的原始人物外观。为了解决这些方面，我们提出了PromptDresser，一个文本可编辑的虚拟试穿模型，利用大型多模态模型（LMM）辅助，基于生成式文本提示实现高质量和多样化的操作。我们的方法通过上下文学习利用LMM，以最小的人力成本为人物和服装图像独立生成详细的文本描述，包括姿势细节和编辑属性。此外，为了确保编辑区域，我们根据文本提示自适应地调整修复掩码。我们的方法增强了文本可编辑性，同时有效传达了仅通过图像难以捕捉的服装细节，从而提高了图像质量。实验表明，PromptDresser显著优于基线，展示了卓越的文本驱动控制和多样的服装操作。我们的代码可在https://github.com/rlawjdghek/PromptDresser获取。
英文摘要：
Recent virtual try-on approaches have advanced by finetuning pre-trained text-to-image diffusion models to leverage their powerful generative ability. However, the use of text prompts in virtual try-on remains underexplored. This paper tackles a text-editable virtual try-on task that modifies the clothing based on the provided clothing image while editing the wearing style (e.g., tucking style, fit) according to the text descriptions. In the text-editable virtual try-on, three key aspects exist: (i) designing rich text descriptions for paired person-clothing data to train the model, (ii) addressing the conflicts where textual information of the existing person's clothing interferes the generation of the new clothing, and (iii) adaptively adjust the inpainting mask aligned with the text descriptions, ensuring proper editing areas while preserving the original person's appearance irrelevant to the new clothing. To address these aspects, we propose PromptDresser, a text-editable virtual try-on model that leverages large multimodal model (LMM) assistance to enable high-quality and versatile manipulation based on generative text prompts. Our approach utilizes LMMs via in-context learning to generate detailed text descriptions for person and clothing images independently, including pose details and editing attributes using minimal human cost. Moreover, to ensure the editing areas, we adjust the inpainting mask depending on the text prompts adaptively. Our approach enhances text editability while effectively conveying clothing details that are difficult to capture through images alone, leading to improved image quality. Experiments show that PromptDresser significantly outperforms baselines, demonstrating superior text-driven control and versatile clothing manipulation. Our code is available at https://github.com/rlawjdghek/PromptDresser.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
