Pretrained large generative language models have shown great performance on many tasks, but exhibit low compositional generalization abilities. Scaling such models has been shown to improve their performance on various NLP tasks even just by conditioning them on a few examples to solve the task without any fine-tuning (also known as in-context learning). In this work, we look at the gap between the in-distribution (ID) and out-of-distribution (OOD) performance of such models in semantic parsing tasks with in-context learning. In the ID settings, the demonstrations are from the same split (test or train) that the model is being evaluated on, and in the OOD settings, they are from the other split. We look at how the relative generalization gap of in-context learning evolves as models are scaled up. We evaluate four model families, OPT, BLOOM, CodeGen and Codex on three semantic parsing datasets, CFQ, SCAN and GeoQuery with different number of exemplars, and observe a trend of decreasing relative generalization gap as models are scaled up.
Does Data Scaling Lead to Visual Compositional Generalization?
Tübingen AI Center, University of Tübingen（图宾根人工智能中心，图宾根大学）；Technical University of Munich（慕尼黑技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Max Planck Institute for Intelligent Systems, Tübingen（马克斯·普朗克智能系统研究所，图宾根）
Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformers
School of Artificial Intelligence, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Institute for Advanced Algorithms Research, Shanghai Seres Information Technology Co., Ltd（上海赛尔斯信息技术有限公司高级算法研究院）
Compositional Instruction Following with Language Models and Reinforcement Learning
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of the Witwatersrand（威特沃特斯兰德大学）；Arizona State University（亚利桑那州立大学）；Georgia Institute of Technology（佐治亚理工学院）
Learning to Substitute Components for Compositional Generalization
School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；Department of Computer Science, Hong Kong Baptist University（香港浸会大学计算机科学系）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）；Beijing Key Laboratory of Research on Large Models and Intelligent Governance（北京大型模型与智能治理研究重点实验室）；Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE（下一代智能搜索与推荐工程研究中心，教育部）；Tianjin University of Science and Technology（天津科技大学）；Beijing Institute of Technology（北京理工大学）
Towards Compositional Generalization of LLMs via Skill Taxonomy Guided Data Synthesis
State Key Laboratory of Complex & Critical Software Environment, Beihang University（复杂与关键软件环境国家重点实验室，北京航空航天大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Beijing Institute of Technology（北京理工大学）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）
Evaluating Morphological Compositional Generalization in Large Language Models
EPFL（瑞士联邦理工学院）；Idiap Research Institute（Idiap研究机构）；Duke University（杜克大学）；Brandeis University（布兰迪斯大学）；LMU Munich（慕尼黑大学）；University of Cambridge（剑桥大学）；Johns Hopkins University（约翰霍普金斯大学）；Università della Svizzera Italiana（瑞士意大利大学）；New York University（纽约大学）
Rule Extrapolation in Language Models: A Study of Compositional Generalization on OOD Prompts
University of Cambridge（剑桥大学）；AI Center, UCL（伦敦大学学院人工智能中心）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ELLIS Institute Tübingen（蒂宾根ELLIS研究所）；Tübingen AI Center（蒂宾根人工智能中心）
ChatGPT as Data Augmentation for Compositional Generalization: A Case Study in Open Intent Detection
Ingenuity Labs Research Institute（创新实验室研究所）；Queen’s University（女王大学）；Smith School of Business（史密斯商学院）；Department of Electrical and Computer Engineering（电气与计算机工程系）
Measuring and Narrowing the Compositionality Gap in Language Models
Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；MosaicML（MosaicML公司）；Meta AI Research（Meta人工智能研究院）；Allen Institute for AI（艾伦人工智能研究所）
Swing-by Dynamics in Concept Learning and Compositional Generalization
CBS-NTT Physics of Intelligence Program, Harvard University（哈佛大学）；Physics & Informatics Laboratories, NTT Research, Inc.（NTT研究公司）；Computer Science and Engineering, University of Michigan（密歇根大学）；Department of Physics, Harvard University（哈佛大学）
一个快速增长的研究机构对合成泛化研究语义解析器的能力，动态重组训练中看到的语言元素到看不见的序列。我们对最近的COGS语料库（Kim和Linzen，2020）上的序列到序列模型和由组成原则指导的模型进行了系统的比较。尽管seq 2seq模型在词汇任务上表现良好，但在需要新语法结构的结构泛化任务上，它们的准确率接近于零;即使它们被训练来预测语法而不是语义，也是如此。相比之下，组合模型在结构泛化方面达到了近乎完美的准确性;我们提出了来自AM解析器的新结果（Groschwitz et al.，2021年）。我们的研究结果表明，结构泛化是组成泛化的一个关键措施，需要知道复杂结构的模型。
Learning to Substitute Spans towards Improving Compositional Generalization
University of Science and Technology of China（中国科学技术大学）；State Key Laboratory of Cognitive Intelligence（认知智能国家重点实验室）；City University of Hong Kong（香港城市大学）
