Self-attention is usually described as a flexible, content-adaptive way to mix a token with information from its past. We reinterpret causal self-attention transformers, the backbone of modern foundation models, within a probabilistic framework, much as classical PCA is extended to probabilistic PCA. This reformulation reveals a key structural consequence of the underlying change of variables: a barrier constraint emerges on the parameters of self-attention. The resulting geometry exposes a degeneracy boundary where the attention-induced mapping becomes locally ill-conditioned, yielding a stability-margin interpretation analogous to the margin in support vector machines. This, in turn, naturally gives rise to the concept of support tokens.
We further show that causal transformers define a consistent stochastic process over infinite token sequences, providing a rigorous probabilistic foundation for sequence modeling. Building on this view, we derive a Bayesian MAP training objective that requires only a minimal modification to standard LLM training: adding a smooth log-barrier penalty to the usual cross-entropy loss. Empirically, the resulting training objective improves robustness to input perturbations and sharpens the margin geometry of the learned representations without sacrificing out-of-sample accuracy.
TOLiD: Bridging the Architecture Gap in Vision Foundation Model to LiDAR Pretraining via Token Lifting for Distillation
SAIVT Group, School of Electrical Engineering and Robotics, Queensland University of Technology（澳大利亚昆士兰科技大学电气工程与机器人学院SAIVT组）；CSIRO Robotics, CSIRO（澳大利亚联邦科学与工业研究组织机器人部）
RmGPT: A Foundation Model with Generative Pre-trained Transformer for Fault Diagnosis and Prognosis in Rotating Machinery
School of Aeronautics and Astronautics, Shanghai Jiao Tong University（航空宇航学院，上海交通大学）；Shanghai Innovation Institute（上海创新研究院）；MINES Paris-PSL University（巴黎-里昂商学院）；Energy Department, Politecnico di Milano（能源部门，米兰理工学院）；Beijing Aeronautical Technology Research Center（北京航空航天科技研究所）
Is Tokenization Needed for Masked Particle Modelling?
University of Geneva（日内瓦大学）；Meta FAIR；Technical University of Munich（慕尼黑工业大学）；SLAC National Accelerator Laboratory（SLAC国家加速器实验室）；Laboratory of Instrumentation and Experimental Particle Physics（仪器与实验粒子物理实验室）；University of Haifa（海法大学）
WLFM: A Well-Logs Foundation Model for Multi-Task and Cross-Well Geological Interpretation
Institute of Advanced Technology, University of Science and Technology of China（科学技术大学先进技术研究所）；Department of Automation, University of Science and Technology of China（科学技术大学自动化系）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合国家科学中心人工智能研究所）
Xi’an Jiaotong University（西安交通大学）；Microsoft Research Asia（微软亚洲研究院）
Obtaining the human-like perception ability of abstracting visual concepts from concrete pixels has always been a fundamental and important target in machine learning research fields such as disentangled representation learning and scene decomposition. Towards this goal, we propose an unsupervised transformer-based Visual Concepts Tokenization framework, dubbed VCT, to perceive an image into a set of disentangled visual concept tokens, with each concept token responding to one type of independent visual concept. Particularly, to obtain these concept tokens, we only use cross-attention to extra…
VFM-VAE: Vision Foundation Models Can Be Good Tokenizers for Latent Diffusion Models
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（人机混合增强智能国家重点实验室，人工智能与机器人研究院，西安交通大学）；Microsoft Research Asia（微软亚洲研究院）
Autonomous labeling of surgical resection margins using a foundation model
University of California, Los Angeles（加州大学洛杉矶分校）；Fatih Sultan Mehmet Vakif University（法提赫·苏丹·穆罕默德·瓦克夫大学）；University of California, Davis（加州大学戴维斯分校）；Hadassah Hebrew University Medical Center（哈达萨希伯来大学医学中心）
Scale-Aware Pretraining of Time Series Foundation Models via Multi-Patch Token Alignment and Hybrid Masking
School of Software, Shandong University（山东大学软件学院）；Joint SDU–NTU Centre for Artificial Intelligence Research (C-FAIR), Shandong University（山东大学-南洋理工大学人工智能联合研究中心（C-FAIR））；Nanyang Technological University（南洋理工大学）
Brain-Semantoks: Learning Semantic Tokens of Brain Dynamics with a Self-Distilled Foundation Model
Hertie Institute for AI in Brain Health, University of Tübingen（图宾根大学脑健康人工智能研究所）；Tübingen AI Center, University of Tübingen（图宾根大学图宾根人工智能中心）；Charité – Universitätsmedizin Berlin, Department of Psychiatry and Psychotherapy（柏林夏里特医学院心理治疗系）
SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
AGI Institute, School of Computer Science, Shanghai Jiao Tong University（AGI研究院，计算机科学学院，上海交通大学）；Shanghai Key Laboratory for Intelligent Information Processing, Fudan University（上海智能信息处理重点实验室，复旦大学）；State Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；School of Artificial Intelligence, Wuhan University（人工智能学院，武汉大学）；Institute of Computer Science, University of Göttingen（计算机科学研究所，哥廷根大学）
Beihang University（北航）；M-A-P；The Hong Kong Polytechnic University（香港理工大学）；AIWaves；University of Alberta（阿尔伯塔大学）；University of Waterloo（滑铁卢大学）；University of Manchester（曼彻斯特大学）；Chinese Academy of Sciences（中国科学院）；Peking University（北京大学）；Shanghai AI Lab（上海AI实验室）；Nanjing University（南京大学）；Kuaishou Technology（快手科技）
LASS-ODE: Scaling ODE Computations to Connect Foundation Models with Dynamical Physical Systems
Department of Electrical, Computer, and Energy Engineering, Arizona State University, AZ, United States（电气、计算机与能源工程系，亚利桑那州立大学）；Air Force Research Laboratory, VA, United States（空军研究实验室）
Graphlets as Building Blocks for Structural Vocabulary in Knowledge Graph Foundation Models
ScaDS.AI Dresden/Leipzig, Technische Universität Dresden（ScaDS.AI 德累斯顿/莱比锡，德累斯顿技术大学）；Department of Mathematics, KNUST（数学系，库马西大学）；alphaspeech - c/o alpha NT GmbH；Amazon, Technische Universität Dresden（亚马逊，德累斯顿技术大学）；TIB – Leibniz Information Centre for Science and Technology（TIB – 莱比锡科学与技术信息中心）
Entropy, Disagreement, and the Limits of Foundation Models in Genomics
Genome Institute of Singapore, A*STAR（新加坡基因组研究院，A*STAR）；KU Leuven（卢森堡大学）；Faculty of Electrical Engineering and Computing, University of Zagreb（扎格雷布大学电子工程与计算学院）
