arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.06338cs.AIcs.CVcs.LG
收藏

扩散变换器中空间关系生成的电路机制

Circuit Mechanisms for Spatial Relation Generation in Diffusion Transformers

Kempner Institute, Harvard University(哈佛大学肯普纳研究所);Harvard University(哈佛大学)

Binxu Wang, Jingxuan Fan, Xu Pan

AI总结:

研究通过机械可解释性方法探讨扩散变换器如何生成正确空间关系,发现不同文本编码器影响电路机制,随机嵌入通过双阶段电路生成,预训练编码器则通过信息融合生成,两种方法在领域内表现相似但抗扰性不同。

摘要:

扩散变换器(DiTs)在文本到图像生成中取得重大进展,但模型仍难以根据文本提示正确生成物体间空间关系。本文采用机械可解释性方法,研究DiT如何生成正确空间关系。我们从头训练不同大小的DiT和不同文本编码器,学习生成包含两个物体的图像,其属性和空间关系由文本提示指定。发现尽管所有模型可近完美完成该任务,但底层机制因文本编码器选择而大幅不同。使用随机文本嵌入时,空间关系信息通过双阶段电路传递,涉及两个交叉注意力头,分别读取文本提示中的空间关系和单个物体属性。使用预训练文本编码器(T5)时,DiT使用不同电路,通过文本令牌的信息融合,共同读取空间关系和单个物体信息。进一步表明,尽管两种设置在领域内表现相似,但对领域外扰动的鲁棒性不同,可能暗示在现实场景中生成正确关系的困难。

英文摘要:

Diffusion Transformers (DiTs) have greatly advanced text-to-image generation, but models still struggle to generate the correct spatial relations between objects as specified in the text prompt. In this study, we adopt a mechanistic interpretability approach to investigate how a DiT can generate correct spatial relations between objects. We train, from scratch, DiTs of different sizes with different text encoders to learn to generate images containing two objects whose attributes and spatial relations are specified in the text prompt. We find that, although all the models can learn this task to near-perfect accuracy, the underlying mechanisms differ drastically depending on the choice of text encoder. When using random text embeddings, we find that the spatial-relation information is passed to image tokens through a two-stage circuit, involving two cross-attention heads that separately read the spatial relation and single-object attributes in the text prompt. When using a pretrained text encoder (T5), we find that the DiT uses a different circuit that leverages information fusion in the text tokens, reading spatial-relation and single-object information together from a single text token. We further show that, although the in-domain performance is similar for the two settings, their robustness to out-of-domain perturbations differs, potentially suggesting the difficulty of generating correct relations in real-world scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑