MMoT：用于组合多模态条件图像合成的多模态标记混合Transformer
MMoT: Mixture-of-Modality-Tokens Transformer for Composed Multimodal Conditional Image Synthesis
AI总结：
提出MMoT模型，通过自适应融合多模态控制信号、平衡训练损失和采样引导，解决组合多模态条件图像合成中的模态协调与不平衡问题，实现高质量忠实图像生成。
摘要：
现有的多模态条件图像合成（MCIS）方法生成图像时，需严格满足所有模态的组合条件，这限制了合成的可控性，并导致跨模态潜力未被充分利用。为此，我们提出在模态不完全互补的组合多模态控制信号条件下生成图像，即组合多模态条件图像合成（CMCIS）。具体而言，我们观察到所提出的CMCIS任务存在两个挑战性问题：模态协调问题和模态不平衡问题。为解决这些问题，我们引入了多模态标记混合Transformer（MMoT），它自适应地融合细粒度的多模态控制信号；一种多模态平衡训练损失，以稳定每个模态的优化；以及一种多模态采样引导，以平衡每个模态控制信号的强度。综合实验结果表明，MMoT在单模态条件图像合成（UCIS）和MCIS任务上均取得了优越性能，能够在复杂多模态条件下生成高质量且忠实的图像。项目网站可在https://jabir-zheng.github.io/MMoT获取。
英文摘要：
Existing multimodal conditional image synthesis (MCIS) methods generate images conditioned on any combinations of various modalities that require all of them must be exactly conformed, hindering the synthesis controllability and leaving the potential of cross-modality under-exploited. To this end, we propose to generate images conditioned on the compositions of multimodal control signals, where modalities are imperfectly complementary, i.e., composed multimodal conditional image synthesis (CMCIS). Specifically, we observe two challenging issues of the proposed CMCIS task, i.e., the modality coordination problem and the modality imbalance problem. To tackle these issues, we introduce a Mixture-of-Modality-Tokens Transformer (MMoT) that adaptively fuses fine-grained multimodal control signals, a multimodal balanced training loss to stabilize the optimization of each modality, and a multimodal sampling guidance to balance the strength of each modality control signal. Comprehensive experimental results demonstrate that MMoT achieves superior performance on both unimodal conditional image synthesis (UCIS) and MCIS tasks with high-quality and faithful image synthesis on complex multimodal conditions. The project website is available at https://jabir-zheng.github.io/MMoT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
