现象空间中的偏斜阻碍了文本到图像生成中的泛化
Skews in the Phenomenon Space Hinder Generalization in Text-to-Image Generation
AI总结：
本文研究文本到图像生成中的泛化问题，提出量化数据集语言和视觉偏斜的统计指标，发现现象覆盖不完整或不平衡直接导致泛化失败，并证明简单分布扰动可在不扩大数据规模下提升泛化。
摘要：
关于文本到图像生成的文献受到如何忠实地将实体与关系组合起来的问题的困扰。但缺乏对如何有效学习实体-关系组合的正式理解。此外，能够有意义地反映问题结构的底层现象空间并未得到良好定义，这导致了追求数据量的军备竞赛，希望泛化能从大规模预训练中涌现。我们假设底层现象覆盖范围并未按比例扩大，导致所呈现现象的偏斜，从而损害了泛化。我们引入了统计指标来量化数据集在关系学习中的语言和视觉偏斜，并表明文本到图像生成的泛化失败是现象覆盖不完整或不平衡的直接结果。我们首先在合成领域进行实验，证明系统控制的指标能强烈预测泛化性能。然后我们转向自然图像，并表明基于我们的理论进行简单的分布扰动可以在不扩大绝对数据规模的情况下提升泛化能力。这项工作指明了一个重要方向，即在不扩大绝对规模的前提下，通过增强数据多样性或平衡性来提升数据质量。我们的讨论指出了重要的开放性问题：1）评估生成的实体-关系组合；2）用于抽象关系推理的更好模型。
英文摘要：
The literature on text-to-image generation is plagued by issues of faithfully composing entities with relations. But there lacks a formal understanding of how entity-relation compositions can be effectively learned. Moreover, the underlying phenomenon space that meaningfully reflects the problem structure is not well-defined, leading to an arms race for larger quantities of data in the hope that generalization emerges out of large-scale pretraining. We hypothesize that the underlying phenomenological coverage has not been proportionally scaled up, leading to a skew of the presented phenomenon which harms generalization. We introduce statistical metrics that quantify both the linguistic and visual skew of a dataset for relational learning, and show that generalization failures of text-to-image generation are a direct result of incomplete or unbalanced phenomenological coverage. We first perform experiments in a synthetic domain and demonstrate that systematically controlled metrics are strongly predictive of generalization performance. Then we move to natural images and show that simple distribution perturbations in light of our theories boost generalization without enlarging the absolute data size. This work informs an important direction towards quality-enhancing the data diversity or balance orthogonal to scaling up the absolute size. Our discussions point out important open questions on 1) Evaluation of generated entity-relation compositions, and 2) Better models for reasoning with abstract relations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
