DiffCloth：通过结构化跨模态语义对齐实现的基于扩散的服装合成与操控
DiffCloth: Diffusion Based Garment Synthesis and Manipulation via Structural Cross-modal Semantic Alignment
AI总结：
针对现有跨模态服装合成方法忽略设计领域图文结构对应的问题，提出DiffCloth扩散流水线，通过部件级二分匹配与语义捆绑交叉注意力实现结构化跨模态对齐，在CM-Fashion基准上取得最优合成效果，还支持区域一致的灵活文本操控。
摘要：
跨模态服装合成与操控将极大助力服装设计师通过灵活的语言交互界面生成服装并修改设计方案。现有方法遵循通用的文本到图像范式，通过简单的交叉注意力模块挖掘跨模态关联，却忽略了服装设计领域中视觉表征与文本表征之间的结构对应关系。\n在本研究中，我们提出了DiffCloth——一种用于跨模态服装合成与操控的基于扩散的流水线，通过结构化对齐跨模态语义，赋予扩散模型在时尚领域的灵活组合能力。具体而言，我们将部件级跨模态对齐建模为语言属性短语（Attribute-Phrases，AP）与视觉服装部件之间的二分图匹配问题，其中视觉服装部件分别通过成分句法分析和语义分割获取。为缓解属性混淆问题，我们进一步提出语义捆绑交叉注意力，以保留每个属性短语中属性形容词与部件名词的注意力图之间的空间结构相似性。此外，DiffCloth支持通过简单替换文本提示中的属性短语来操控生成结果，与操控无关的区域会通过属性短语的捆绑注意力图得到的混合掩码被识别并保持不变。\n在CM-Fashion基准上的大量实验表明，DiffCloth既通过利用固有结构信息取得了当前最优的服装合成结果，也支持具备区域一致性的灵活操控。
英文摘要：
Cross-modal garment synthesis and manipulation will significantly benefit the way fashion designers generate garments and modify their designs via flexible linguistic interfaces.Current approaches follow the general text-to-image paradigm and mine cross-modal relations via simple cross-attention modules, neglecting the structural correspondence between visual and textual representations in the fashion design domain. In this work, we instead introduce DiffCloth, a diffusion-based pipeline for cross-modal garment synthesis and manipulation, which empowers diffusion models with flexible compositionality in the fashion domain by structurally aligning the cross-modal semantics. Specifically, we formulate the part-level cross-modal alignment as a bipartite matching problem between the linguistic Attribute-Phrases (AP) and the visual garment parts which are obtained via constituency parsing and semantic segmentation, respectively. To mitigate the issue of attribute confusion, we further propose a semantic-bundled cross-attention to preserve the spatial structure similarities between the attention maps of attribute adjectives and part nouns in each AP. Moreover, DiffCloth allows for manipulation of the generated results by simply replacing APs in the text prompts. The manipulation-irrelevant regions are recognized by blended masks obtained from the bundled attention maps of the APs and kept unchanged. Extensive experiments on the CM-Fashion benchmark demonstrate that DiffCloth both yields state-of-the-art garment synthesis results by leveraging the inherent structural information and supports flexible manipulation with region consistency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
