Foundation models, particularly Large Language Models (LLMs), have revolutionized text and video processing, yet time series data presents distinct challenges for such approaches due to domain-specific features such as missing values, multi-resolution characteristics, etc. Furthermore, the de-facto autoregressive transformers tend to learn deterministic temporal dependencies within pre-trained data while overlooking inherent uncertainties and lacking integration of physical constraints. In this paper, we introduce TimeDiT, a diffusion transformer model that synergistically combines transformer-based temporal dependency learning with diffusion-based probabilistic sampling. TimeDiT employs a unified masking mechanism to harmonize the training and inference process across diverse tasks while introducing a theoretically grounded, finetuning-free model editing strategy that enables flexible integration of external knowledge during sampling. Acknowledging the challenges of unifying multiple downstream tasks under a single model, our systematic evaluation demonstrates TimeDiT's effectiveness both in fundamental tasks, i.e., forecasting and imputation, through zero-shot/fine-tuning; and in domain tasks, i.e., multi-resolution forecasting, anomaly detection, and data generation, establishing it as a \textit{proto-foundation model} that bridges the gap between general-purpose and domain-specific models.
Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting
Morgan Stanley（摩根士丹利）；ServiceNow Research（ServiceNow 研究院）；Université de Montréal（蒙特利尔大学）；Mila-Quebec AI Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）
University of Notre Dame（诺丁汉大学）；University of Connecticut（康涅狄格大学）；University of Virginia（弗吉尼亚大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Cambridge（剑桥大学）；Mila - Québec AI Institute（魁北克AI研究院）；Université de Montréal（蒙特利尔大学）
Arizona State University（亚利桑那州立大学）；Purdue University（普渡大学）；University of British Columbia（不列颠哥伦比亚大学）；Stanford University（斯坦福大学）；The Ohio State University（俄亥俄州立大学）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Virginia Polytechnic Institute（弗吉尼亚多项技术学院）；University of Michigan（密歇根大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Nanjing University（南京大学）
Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis
CMU（卡内基梅隆大学）；Bosch Center for AI（博世人工智能中心）；MIT（麻省理工学院）；Field AI（菲尔德人工智能公司）；Georgia Tech（佐治亚理工学院）；FAIR at Meta（元公司人工智能研究院）；UC San Diego（加利福尼亚大学圣迭戈分校）；SAIR Lab（空间人工智能与机器人实验室）；Google DeepMind（谷歌DeepMind）
CDFM: Towards a General-Purpose Causal Discovery Foundation Model
School of Computer Science, Guangdong University of Technology, Guangzhou, China（广东技术大学计算机科学学院）；College of Mathematics and Computer, Shantou University, Shantou, China（汕头大学数学与计算机学院）；Department of Computer Science and Technology, Tsinghua University, Beijing, China（清华大学计算机科学与技术系）
Are foundation models useful feature extractors for electroencephalography analysis?
Chair for AI in Healthcare and Medicine, Technical University of Munich（人工智能在医疗与健康中的研究所，慕尼黑技术大学）；Department of Neurology, Technical University of Munich（神经病学系，慕尼黑技术大学）；Center for Interdisciplinary Pain Medicine, Technical University of Munich（跨学科疼痛医学中心，慕尼黑技术大学）；Munich Center for Machine Learning, Munich, Germany（慕尼黑机器学习中心，德国慕尼黑）；Department of Computing, Imperial College London（计算系，伦敦帝国学院）
Beijing University of Posts and Telecommunications（北京邮电大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；West China Hospital, Sichuan University（四川大学华西医院）；Shanghai Jiao Tong University（上海交通大学）；Sichuan University Pittsburgh Institute（四川大学匹兹堡学院）
Inevitable domain and task discrepancies in real-world scenarios can impair the generalization performance of the pre-trained deep models for medical data. Therefore, we audaciously propose that we should build a general-purpose medical AI system that can be seamlessly adapted to downstream domains/tasks. Since the domain/task adaption procedures usually involve additional labeling work for the target data, designing a data-efficient adaption algorithm is desired to save the cost of transferring the learned knowledge. Our recent work found that vision-language models (VLMs) are efficient learn…
CEHR-XGPT: A Scalable Multi-Task Foundation Model for Electronic Health Records
Columbia University Irving Medical Center（哥伦比亚大学伊万格琳医疗中心）；Observational Health Data Sciences and Informatics（观察性健康数据科学与信息学）；Department of Biomedical Informatics（生物医学信息学系）；Columbia University（哥伦比亚大学）；NewYork-Presbyterian Hospital（纽约- Presbyterian医院）；Department of Computer Science（计算机科学系）
Towards a general-purpose foundation model for fMRI analysis
The Chinese University of Hong Kong, Hong Kong（香港中文大学）；Yonsei University, Seoul, South Korea（延世大学）；University of Georgia, Athens, GA（佐治亚大学）；Peking University Health Science Center, Beijing, China（北京大学医学部）；University of Sydney, Sydney, Australia（悉尼大学）；Yale School of Medicine, New Haven, CT（耶鲁医学院）；Emory University, Atlanta, GA（埃默里大学）；Lehigh University, Bethlehem, PA（莱斯利大学）；Boston Children’s Hospital, Boston, MA（波士顿儿童医院）；Massachusetts General Hospital, Boston, MA（麻省总医院）；Brigham and Women’s Hospital, Boston, MA（布里奇沃特医院）；Hong Kong Polytechnic University, Hong Kong（香港理工大学）；Kempner Institute for Natural and Artificial Intelligence, Cambridge, MA（Kempner自然与人工智能研究所）
FLOWR.root: A flow matching based foundation model for joint multi-purpose structure-aware 3D ligand generation and affinity prediction
Machine Learning & Computational Sciences, Pfizer Worldwide R&D（辉瑞全球研发 机器学习与计算科学部）；Computational Chemistry, Medicine Design, Pfizer Worldwide R&D（辉瑞全球研发 计算化学与医学设计部）；Doctoral School of Medical and Health Sciences, Jagiellonian University Medical College（杰拉西利昂大学医学院医学与健康科学博士学院）；Department of Physicochemical Drug Analysis, Faculty of Pharmacy, Jagiellonian University Medical College（杰拉西利昂大学医学院药物物理化学分析系）；Institute of Structural Biology, Molecular Targets and Therapeutics Center, Helmholtz Munich（海德堡-慕尼黑分子靶点与治疗中心 结构生物学研究所）；TUM School of Natural Sciences, Department of Bioscience, Bayerisches NMR Zentrum, Technical University of Munich（慕尼黑技术大学自然科学学院 生物科学系，拜耳核磁共振中心）
A big convergence of model architectures across language, vision, speech, and multimodal is emerging. However, under the same name "Transformers", the above areas use different implementations for better performance, e.g., Post-LayerNorm for BERT, and Pre-LayerNorm for GPT and vision Transformers. We call for the development of Foundation Transformer for true general-purpose modeling, which serves as a go-to architecture for various tasks and modalities with guaranteed training stability. In this work, we introduce a Transformer variant, named Magneto, to fulfill the goal. Specifically, we pro…
