图像的反馈：通过抑制图像到文本的反馈来引导多模态扩散变换器（MMDiTs）
Images Talk Back: Guiding MMDiTs by Suppressing Image-to-Text Feedback
AI总结：
该研究针对MMDiTs的图像到文本反馈问题，提出SFG及原位变体iSFG方法，在图像、视频生成任务上提升了对齐度与偏好评分。
摘要：
多模态扩散变换器（MMDiTs）允许演化中的图像通过联合注意力更新内部文本表示，这些表示随后会引导后续的图像更新，由此产生一个问题：这种反馈是有助于纠正与提示的偏差，还是会保留偏差？我们发现，图像到文本（I2T）反馈能够保留演化图像中已表示的对象和属性，包括错误；移植I2T消息可以修复失败案例或破坏成功案例。在匹配的干预强度下，抑制I2T反馈的影响会提升生成质量，而相反的控制操作则会降低生成质量。这些发现催生了语义反馈引导（SFG）方法，该方法通过引导远离会增强I2T反馈并削弱文本到图像条件作用的辅助分支，来抑制I2T反馈的影响；其原位变体iSFG直接抑制I2T反馈并增强文本到图像条件作用，无需额外的模型评估。时间干预表明，在图像采样步骤的前20%运行SFG分支即可，这在无无分类器引导（CFG）的情况下仅需约1.2倍的基础NFE。在三个图像生成主干上的实验显示，对齐度和偏好模型评分均有提升：在FLUX上，将SFG添加到CFG中使GenEval从0.6661提升至0.7479；在SD3.5M上，iSFG在零额外NFE的情况下，相比CFG使HPSv2.1提升了1.33；该控制还可迁移到文本到视频和图像到视频生成，提升VBench评分。代码可在该https URL获取。
英文摘要：
Multimodal diffusion transformers (MMDiTs) allow the evolving image to update internal text representations through joint attention. These representations then guide subsequent image updates, raising a question: does this feedback help correct deviations from the prompt, or preserve them? We find that image-to-text (I2T) feedback can preserve the objects and attributes already represented in the evolving image, including errors. Transplanting I2T messages can rescue failures or disrupt successes. At matched intervention strengths, suppressing the influence of I2T feedback improves generation, whereas opposite controls degrade it. These findings motivate Semantic Feedback Guidance (SFG), which suppresses the influence of I2T feedback by guiding away from an auxiliary branch that strengthens this feedback and weakens text-to-image conditioning. The in-place variant, iSFG, directly suppresses I2T feedback and strengthens text-to-image conditioning, requiring no extra model evaluations. Temporal interventions motivate running the SFG branch during the first 20% of image sampling steps, requiring about $1.2\times$ the base NFE without classifier-free guidance (CFG). Experiments across three image-generation backbones show gains in alignment and preference-model scores. On FLUX, adding SFG to CFG raises GenEval from $0.6661$ to $0.7479$; on SD3.5M, iSFG adds $+1.33$ HPSv2.1 over CFG at zero extra NFE. The control also transfers to text-to-video and image-to-video generation, improving VBench scores. Code is available at https://github.com/yyc1215lll-creator/SFG.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
