类型多样性使Transformer能够进行组合泛化
Type Diversity Enables Transformers to Generalise Compositionally
- Aalto University（阿尔托大学）
- University of Helsinki（赫尔辛基大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本研究提出类型多样性（而非Transformer固有缺陷）是导致结构泛化比词汇泛化更困难的原因，并通过创建COGS和SLOG的语言学多样化变体验证了该假设，同时反驳了复合发散解释，并探讨了其他数据集属性的影响。
AI中文摘要：
组合泛化已被划分为词汇泛化和结构泛化。先前的研究发现，对于Transformer而言，结构泛化比词汇泛化更难。我们提出，这种差异并非Transformer固有的，而是由于先前工作中特定数据集中词汇类型的高多样性和结构类型的低多样性所致。这里所说的类型多样性，是指该类型的不同构造器的数量，而非例如可能填充该结构的特定单词组合。为了验证这一点，我们在先前发布的数据集中改变词汇和结构类型的多样性数量。我们使用语法框架创建了COGS和SLOG数据集的语言学多样化变体。我们发现，类型多样性与词汇和结构测试用例中的组合泛化相关性相同，这支持了我们的假设。我们注意到，这与先前工作中关于复合发散解释了组合泛化任务难度的命题相矛盾。我们进一步研究了其他数据集属性对组合泛化的影响，例如新颖测试结构之外的其他类型的多样性，以及逻辑语义格式的表面属性。
英文摘要：
Compositional generalisation has been divided into lexical and structural generalisation. Previous work has found that structural generalisation is harder than lexical for Transformers. We propose that this difference is not inherent to Transformers, but due to the high diversity of lexical types and low diversity of structural types in the specific datasets of these previous works. By type diversity we mean the number of different constructors of that type, instead of, for example, the specific word combinations that might populate the structure. To test this, we vary the amounts of type diversity of lexical and structural types in previously published datasets. We create linguistically diverse variants of the COGS and SLOG datasets using Grammatical Framework. We find that type diversity correlates with compositional generalisation equally in lexical and structural test cases, supporting our hypothesis. We note a contradiction with the proposition in previous work that compound divergence explains the difficulty in compositional generalisation tasks. We further investigate the effects of other dataset properties on compositional generalisation, such as the diversity of types other than the novel test structure, and surface properties of the logical semantics format.
