Large Pre-trained Language Models (PLM) have become the most desirable starting point in the field of NLP, as they have become remarkably good at solving many individual tasks. Despite such success, in this paper, we argue that current paradigms of working with PLMs are neglecting a critical aspect of modeling human intelligence: functional compositionality. Functional compositionality - the ability to compose learned tasks - has been a long-standing challenge in the field of AI (and many other fields) as it is considered one of the hallmarks of human intelligence. An illustrative example of such is cross-lingual summarization, where a bilingual person (English-French) could directly summarize an English document into French sentences without having to translate the English document or summary into French explicitly. We discuss why this matter is an important open problem that requires further attention from the field. Then, we show that current PLMs (e.g., GPT-2 and T5) don't have functional compositionality yet and it is far from human-level generalizability. Finally, we suggest several research directions that could push the field towards zero-shot functional compositionality of language models.
Augmenters at SemEval-2023 Task 1: Enhancing CLIP in Handling Compositionality and Ambiguity for Zero-Shot Visual WSD through Prompt Augmentation and Text-To-Image Diffusion
University of Maryland, College Park(马里兰大学帕克分校);Duolingo, Inc.(多邻国公司)
Exploring the Compositional Deficiency of Large Language Models in Mathematical Reasoning
School of Computer Science, Fudan University(复旦大学计算机科学学院);Shanghai Key Laboratory of Intelligent Information Processing, Fudan University(复旦大学上海智能信息处理重点实验室)
How do Transformer Embeddings Represent Compositions? A Functional Analysis
ASUS Intelligent Cloud Services(ASUS智能云服务);Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)(高性能计算研究所(A*STAR));Center for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR)(前沿人工智能研究中心(A*STAR));Texas A&M University(德克萨斯大学阿姆斯特朗分校);McGill University(麦吉尔大学)
Zero-Shot Generalization using Intrinsically Motivated Compositional Emergent Protocols
Indian Institute of Science(印度科学研究所)
人类语言被描述为一个制造\textit{use of finite means to express an unlimited array of thoughts}的系统。特别感兴趣的是组合性方面,由此,复合语言表达的含义可以从其组成部分的含义中推导出来。如果人工智能体能够开发出类似于人类语言的组合通信协议,它们就可以无缝地推广到看不见的组合。研究已经认识到好奇心在促进儿童语言发展方面的作用。在本文中,我们试图使用这种内在的反馈,诱导一个系统的和明确的原语言。我们演示了组合性如何使代理不仅与看不见的对象进行交互,而且还将技能从一个任务转移到另一个任务中的zero-shot设置:\textit{Can an agent, trained to `pull' and `push twice', `pull twice'?}。
Improving Task Generalization via Unified Schema Prompt
Sun Yat-sen University(中山大学);Chinese University of Hong Kong(香港中文大学);Tsinghua University(清华大学);Langboat Technology(澜舟科技);Microsoft Research Asia(微软亚洲研究院)
LLM-based MOFs Synthesis Condition Extraction using Few-Shot Demonstrations
School of Computer Science, Beihang University(北京航空航天大学计算机学院);School of Materials Science & Engineering, University of Science and Technology Beijing(北京科技大学材料科学与工程学院);Westlake University(西湖大学)
Compositional Reasoning Depth Predicts Clinical AI Failure: Empirical Evidence Consistent with Transformer Compositionality Limits in Electronic Health Record Question Answering
University of California San Francisco(加州大学旧金山分校);Waymark
Internal and external pressures on language emergence: least effort, object constancy and frequency
University of Amsterdam(阿姆斯特丹大学);LTL, University of Cambridge(剑桥大学语言技术实验室);ILLC, University of Amsterdam(阿姆斯特丹大学逻辑、语言与计算研究所);Universitat Pompeu Fabra(庞培法布拉大学)