In order to understand the in-context learning phenomenon, recent works have adopted a stylized experimental framework and demonstrated that Transformers can learn gradient-based learning algorithms for various classes of real-valued functions. However, the limitations of Transformers in implementing learning algorithms, and their ability to learn other forms of algorithms are not well understood. Additionally, the degree to which these capabilities are confined to attention-based models is unclear. Furthermore, it remains to be seen whether the insights derived from these stylized settings can be extrapolated to pretrained Large Language Models (LLMs). In this work, we take a step towards answering these questions by demonstrating the following: (a) On a test-bed with a variety of Boolean function classes, we find that Transformers can nearly match the optimal learning algorithm for 'simpler' tasks, while their performance deteriorates on more 'complex' tasks. Additionally, we find that certain attention-free models perform (almost) identically to Transformers on a range of tasks. (b) When provided a teaching sequence, i.e. a set of examples that uniquely identifies a function in a class, we show that Transformers learn more sample-efficiently. Interestingly, our results show that Transformers can learn to implement two distinct algorithms to solve a single task, and can adaptively select the more sample-efficient algorithm depending on the sequence of in-context examples. (c) Lastly, we show that extant LLMs, e.g. LLaMA-2, GPT-4, can compete with nearest-neighbor baselines on prediction tasks that are guaranteed to not be in their training set.
Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey
State Key Lab of Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);School of Computing and Mathematical Sciences, Birkbeck, University of London(伦敦大学伯克贝克学院计算与数学科学学院)
LLMs Explain't: A Post-Mortem on Semantic Interpretability in Transformer Models
Distributed Operating Systems Group, Department of Informatics, Universität Hamburg, Germany(信息学院分布式操作系统组,乌尔姆大学);Machine Learning Group, Department of Informatics, Universität Hamburg, Germany(信息学院机器学习组,乌尔姆大学)
Fusing Large Language Models with Temporal Transformers for Time Series Forecasting
University of Science and Technology of China(中国科学技术大学);University of Washington(华盛顿大学);Beijing Northern Computility InterConnection Co., Ltd.(北京北方计算互联有限公司);ENN Group Co., Ltd.(ENN集团有限公司)
A Short Survey of Human Mobility Prediction in Epidemic Modeling from Transformers to LLMs
Groupe de Recherche de Prospection et Valorisation des Données (Greprovad)(数据勘探与增值研究组(Greprovad));GRIC;Université de Sherbrooke(谢布鲁克大学);University of Kinshasa(金沙萨大学);Montreal Behavioural Medicine Centre(蒙特利尔行为医学中心);Centre Intégré Universitaire de Santé et Services Sociaux du Nord-de-l’Île-de-Montréal (CIUSSS-NIM)(蒙特利尔岛北部综合大学卫生与社会服务中心(CIUSSS-NIM));LISV-UVSQ;Université Paris-Saclay(巴黎-萨克雷大学);University of Klagenfurt(克拉根福大学);Biomedical Research Unit, Hospital Monkole(蒙科尔医院生物医学研究单元);University of Florida(佛罗里达大学);School of Electrical Engineering and Computer Science, University of Ottawa(渥太华大学电气工程与计算机科学学院);Centre National d’Intelligence Épidémiologique (CNIEP)(国家流行病学情报中心(CNIEP))
Bridging the Dimensional Chasm: Uncover Layer-wise Dimensional Reduction in Transformers through Token Correlation
School of Physics, Peking University(北京大学物理学院);Institute of Theoretical Physics, Chinese Academy of Sciences(中国科学院理论物理研究所);Center for High Energy Physics, Peking University(北京大学高能物理研究中心);Beijing Computational Science Research Center(北京计算科学研究中心)
Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with Transformers
National University of Singapore(新加坡国立大学);Institute of Data Science, National University of Singapore(新加坡国立大学数据科学研究院);Integrative Sciences and Engineering Programme, National University of Singapore(新加坡国立大学整合科学与工程计划(注:该计划为新加坡国立大学设立的跨学科项目,此处保留机构相关表述));Tencent(腾讯);Massachusetts Institute of Technology(麻省理工学院);Guangdong Lab of AI and Digital Economy (SZ)(广东省人工智能与数字经济实验室(深圳))
Transformer-Assisted LLM-Based Source Code Summarisation: to Enable More Secure Software Development
School of Computing and Communications, Lancaster University, UK(兰卡斯特大学计算与通讯学院);College of Engineering & Computer Science, VinUniversity(Vin大学工程与计算机科学学院)
Faith and Fate: Limits of Transformers on Compositionality
Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学);University of Southern California(南加州大学);University of Chicago(芝加哥大学)
RecurFormer: Not All Transformer Heads Need Self-Attention
Nanyang Technological University(南洋理工大学);The University of New South Wales(新南威尔士大学);Shanghai Jiao Tong University(上海交通大学);The College of William and Mary(威廉玛丽学院)
How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMs
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University(通用人工智能国家重点实验室,智能科学与技术学院,北京大学);School of EECS, Peking University(电子工程系,北京大学);Huawei Noah’s Ark Lab(华为诺亚实验室);Center for Machine Learning Research, Peking University(机器学习研究中心,北京大学)
Interpreting token compositionality in LLMs: A robustness analysis
Department of Computer Science, University of Manchester(曼彻斯特大学计算机科学系);Idiap Research Institute(Idiap研究机构);National Biomarker Centre, CRUK-MI, Univ. of Manchester(国家生物标记中心,CRUK-MI,曼彻斯特大学)
Case-Based or Rule-Based: How Do Transformers Do the Math?
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);National Key Laboratory of General Artificial Intelligence, BIGAI(北京智源人工智能研究院通用人工智能国家重点实验室)