What is the relationship between model architecture and the ability to perform in-context learning? In this empirical study, we take the first steps toward answering this question. We evaluate thirteen model architectures capable of causal language modeling across a suite of synthetic in-context learning tasks. These selected architectures represent a broad range of paradigms, including recurrent and convolution-based neural networks, transformers, state space model inspired, and other emerging attention alternatives. We discover that all the considered architectures can perform in-context learning under a wider range of conditions than previously documented. Additionally, we observe stark differences in statistical efficiency and consistency by varying the number of in-context examples and task difficulty. We also measure each architecture's predisposition towards in-context learning when presented with the option to memorize rather than leverage in-context examples. Finally, and somewhat surprisingly, we find that several attention alternatives are sometimes competitive with or better in-context learners than transformers. However, no single architecture demonstrates consistency across all tasks, with performance either plateauing or declining when confronted with a significantly larger number of in-context examples than those encountered during gradient-based training.
Provable In-Context Learning of Nonlinear Regression with Transformers
Department of ECE（电子工程系）；The Ohio State University（俄亥俄州立大学）；Engineering Systems and Design Pillar（工程系统与设计支柱）；Singapore University of Technology and Design（新加坡科技设计大学）
In-Context Learning for Non-Stationary MIMO Equalization
Department of Computer Science and Engineering, The Ohio State University（计算机科学与工程系，俄亥俄州立大学）；Michigan Institute for Computational Discovery and Engineering, University of Michigan（密歇根大学计算发现与工程研究所）；Department of Electrical Engineering and Computer Science, University of Michigan（电气工程与计算机科学系，密歇根大学）；Department of Statistics, University of Michigan（统计学系，密歇根大学）
ICLAD: In-Context Learning for Unified Tabular Anomaly Detection Across Supervision Regimes
Department of Electrical and Computer Engineering, McGill University, Canada（麦吉尔大学电气与计算机工程系，加拿大）；Mila - Quebec Artificial Intelligence Institute, Montreal, Canada（魁北克人工智能研究所，蒙特利尔，加拿大）
Exploring Imbalanced Annotations for Effective In-Context Learning
Department of Statistics and Data Science, Southern University of Science and Technology（南方科技大学统计与数据科学系）；School of Economics and Finance, Xi’an Jiaotong University（西安交通大学经济金融学院）；School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学学院）
Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?
LMU Munich（慕尼黑大学）；Siemens AG（西门子公司）；Technical University of Munich（慕尼黑工业大学）；University of Oxford（牛津大学）；University of California, Santa Barbara（加州大学圣巴巴拉分校）
Provable Low-Frequency Bias of In-Context Learning of Representations
Computer Science and Engineering, University of Michigan（密歇根大学计算机科学与工程系）；CBS-NTT Physics of Intelligence Program, Harvard University（哈佛大学CBS-NTT智能物理计划）；Physics of Artificial Intelligence Group, NTT Research, Inc.（NTT研究公司人工智能物理小组）
SMMILE: An Expert-Driven Benchmark for Multimodal Medical In-Context Learning
ETH Zurich（苏黎世联邦理工学院）；Stanford University（斯坦福大学）；Lund University（隆德大学）；Jawaharlal Institute of Postgraduate Medical Education and Research（贾瓦哈拉尔·尼赫鲁医学教育与研究学院）；UCSF（加州大学旧金山分校）；University of Zurich（苏黎世大学）；University Hospital Zurich（苏黎世大学医院）；HOPPR
The Effect of Training Task Diversity on In-Context Learning through the Lens of Low-Dimensional Subspaces
University of California, Berkeley（加州大学伯克利分校）；University of Washington（华盛顿大学）；University of California, Los Angeles（加州大学洛杉矶分校）；Stanford University（斯坦福大学）；University of Toronto（多伦多大学）
