Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing
Gatsby Computational Neuroscience Unit（Gatsby计算神经科学单元）；University College London（伦敦大学学院）；Institute of Science and Technology Austria（奥地利科学与技术研究所）；Sainsbury Wellcome Centre（萨金斯-韦尔科姆中心）
Valentina Njaradi, Clémentine Dominé, Rachel Swanson, Marco Mondelli, Andrew Saxe
Learning to generalise from limited data is a fundamental challenge for both artificial and biological systems. A common strategy is to extract reusable structure from abundant unlabelled data, enabling efficient adaptation to new tasks from limited labelled data. This two-stage paradigm is now standard in modern training pipelines, where pretraining is followed by fine-tuning or linear probing. We provide an analytical model of this process: structure extraction is formalized as principal component analysis on unlabelled data, and downstream learning as linear regression on a separate labelled dataset. In the high-dimensional regime, we derive exact expressions for training and generalisation error showcasing their dependence on representation dimensionality, unlabelled and labelled sample sizes, and task alignment. Our results show that pretrained representations strongly influence downstream generalisation, and we characterize the optimal representation size as a function of task parameters: with abundant pretraining data but scarce downstream data, maximally compressed representations are optimal, whereas with limited pretraining data, higher-dimensional representations generalise better. Furthermore, we establish an exact trade-off between pretraining and supervision, quantifying how much unlabelled data is required to replace a single labelled sample. Beyond our idealised model, we observe similar phenomenology in autoencoders and pretrained LLMs. Altogether, we highlight that optimising representation size is critical, giving conditions for when compression during pretraining improves generalisation.
GeoNIRS: Geometry-Aware Pretraining for Heterogeneous fNIRS Representation Learning
Southern University of Science and Technology（南方科技大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Omni-Intelligence（奥米智能）；Shenzhen Loop Area Institute（深圳河套学院）
Transductive Linear Probing: A Novel Framework for Few-Shot Node Classification
Arizona State University（亚利桑那州立大学）；University of Virginia（弗吉尼亚大学）
Few-Shot节点分类的任务是提供准确的预测节点从新的类只有几个代表性的标记节点。这个问题已经引起了极大的关注，因为它的投影到流行的现实世界的应用，如产品分类的新添加的商品类别的电子商务平台上的稀缺记录或罕见疾病的诊断上的患者相似性图。为了解决非欧几里德图域中具有挑战性的标签稀缺问题，元学习已经成为一种成功和主导的范式。最近，受图自监督学习发展的启发，为Few-Shot节点分类转移预训练的节点嵌入可能是元学习的一个有前途的替代方案，但尚未公开。在这项工作中，我们经验性地证明了另一种框架\textit{Transductive Linear Probing}的潜力，该框架可以传输从图对比学习方法中学习的预训练节点嵌入。我们进一步将Few-Shot节点分类的设置从标准的完全监督扩展到更现实的自监督设置，其中由于缺乏来自训练类的监督，元学习方法无法轻松部署。令人惊讶的是，即使没有任何地面事实标签，具有自监督图对比预训练的直推线性探测也可以在相同协议下优于最先进的完全监督元学习方法。我们希望这项工作可以揭示新的光在Few-Shot节点分类问题，并促进未来的研究，从几乎没有标记的实例图的学习。
A Controlled Counterexample to Strong Proxy-Based Explanations of OOD Performance: in a Fixed Pretraining-and-Probing Setup
School of Life Science and Technology, Institute of Science Tokyo（生命科学与技术学院，科学东京研究所）；Department of Computational Biology and Medical Sciences, Graduate School of Frontier Sciences（计算生物学与医学科学系，前沿科学研究生院）
Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classification
University of Kassel（卡塞尔大学）；Fraunhofer IEE（弗劳恩霍夫研究所）；Ghent University（根特大学）；ML and Systems Biology, MPI of Biochemistry（生物化学Max Planck研究所）；INRIA Montpellier（蒙彼利埃INRIA）
New York University（纽约大学）；Google（谷歌）；Stanford University（斯坦福大学）；NVIDIA（英伟达）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Baylor College of Medicine（贝勒医学院）；Rice University（莱斯大学）；Columbia University（哥伦比亚大学）；Canadian Institute for Advanced Research（加拿大高等研究院）
REVE: A Foundation Model for EEG -- Adapting to Any Setup with Large-Scale Pretraining on 25,000 Subjects
IMT Atlantique Lab-STICC UMR CNRS 6285（IMT Atlantique 实验室）；Psychology Department Université de Montréal（蒙特利尔大学心理学系）；Mila (Quebec AI research institute)（魁北克人工智能研究院）；UNIQUE (Quebec Neuro-AI research center)（魁北克神经人工智能研究中心）
