While pre-trained language models (PTLMs) have achieved noticeable success on many NLP tasks, they still struggle for tasks that require event temporal reasoning, which is essential for event-centric applications. We present a continual pre-training approach that equips PTLMs with targeted knowledge about event temporal relations. We design self-supervised learning objectives to recover masked-out event and temporal indicators and to discriminate sentences from their corrupted counterparts (where event or temporal indicators got replaced). By further pre-training a PTLM with these objectives jointly, we reinforce its attention to event and temporal information, yielding enhanced capability on event temporal reasoning. This effective continual pre-training framework for event temporal reasoning (ECONET) improves the PTLMs' fine-tuning performances across five relation extraction and question answering tasks and achieves new or on-par state-of-the-art performances in most of our downstream tasks.
AWS AI Labs（AWS人工智能实验室）；University of Southern California（南加州大学）
Pretrained language models (PTLMs) are typically learned over a large, static corpus and further fine-tuned for various downstream tasks. However, when deployed in the real world, a PTLM-based model must deal with data distributions that deviate from what the PTLM was initially trained on. In this paper, we study a lifelong language model pretraining challenge where a PTLM is continually updated so as to adapt to emerging data. Over a domain-incremental research paper stream and a chronologically-ordered tweet stream, we incrementally pretrain a PTLM with different continual learning algorithm…
ELO: Efficient Layer-Specific Optimization for Continual Pretraining of Multilingual LLMs
Seoul National University of Science and Technology（首尔科学技术大学）；LG CNS；Korea Advanced Institute of Science and Technology（韩国科学技术院）；Hanbat National University（汉 bat 国立大学）
Domain-Adaptive Pretraining Enhances Water Treatment Semantic Representation for Large-Scale Structured Literature Mining
UNSW Water Research Centre, School of Civil and Environmental Engineering, The University of New South Wales（新南威尔士大学）；School of Electrical Engineering and Computer Science, The University of Queensland（昆士兰大学）；Microsoft Copilot Studio AI（微软Copilot Studio AI）；Departments of Mathematics & Department of Computer and Information Science, University of Pennsylvania（宾夕法尼亚大学）；Department of Civil Engineering, The University of Hong Kong（香港大学）
Continual Pretraining on Encrypted Synthetic Data for Privacy-Preserving LLMs
The PII shown in the figure is synthetic and not real（合成的PII）；International Digital Economy Academy（国际数字经济学院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；The Hong Kong University of Science and Technology（香港科学与技术大学）；DataArc Tech Ltd（DataArc科技有限公司）
A Practitioner's Guide to Continual Multimodal Pretraining
Tübingen AI Center（蒂宾根人工智能中心）；University of Tübingen（蒂宾根大学）；Helmholtz Munich（亥姆霍兹慕尼黑中心）；University of Cambridge（剑桥大学）；LAION；Jülich Supercomputing Center (JSC)（于利希超级计算中心）；Research Center Jülich (FZJ)（于利希研究中心）；Helmholtz Association（亥姆霍兹联合会）；Google DeepMind（谷歌DeepMind）；Munich Center for ML（慕尼黑机器学习中心）；Technical University of Munich（慕尼黑工业大学）
RING: Retrieval-Internalized Generation for Continual Large-Scale Knowledge Injection
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS（中国科学院计算技术研究所人工智能安全国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Xiaohongshu Inc.（小红书科技有限公司）
Munsit at NADI 2025 Shared Task 2: Pushing the Boundaries of Multidialectal Arabic ASR with Weakly Supervised Pretraining and Continual Supervised Fine-tuning
Data Engineering for Scaling Language Models to 128K Context
University of Edinburgh（爱丁堡大学）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）；University of Melbourne（墨尔本大学）；Ohio State University（俄亥俄州立大学）；University of Washington（华盛顿大学）；MIT（麻省理工学院）；UIUC（伊利诺伊大学香槟分校）
Microsoft Research（微软研究院）；Zhejiang University（浙江大学）；Westlake University（西湖大学）；Westlake Institute for Advanced Study（西湖高等研究院）
Prompt-based learning, with its capability to tackle zero-shot and few-shot NLP tasks, has gained much attention in community. The main idea is to bridge the gap between NLP downstream tasks and language modeling (LM), by mapping these tasks into natural language prompts, which are then filled by pre-trained language models (PLMs). However, for prompt learning, there are still two salient gaps between NLP tasks and pretraining. First, prompt information is not necessarily sufficiently present during LM pretraining. Second, task-specific data are not necessarily well represented during pretrain…
Center for Information and Language Processing, LMU Munich（信息与语言处理中心，慕尼黑莱茵河大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心（MCML））；Bosch Center for Artificial Intelligence（博世人工智能中心）
Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this paper, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language m…
LLMs Can Evolve Continually on Modality for X-Modal Reasoning
Dalian University of Technology（大连理工大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Tsinghua University（清华大学）；The Hong Kong University of Science and Technology（香港科技大学）
Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale
Chinese Academy of Sciences（中国科学院）；Harbin Institute of Technology（哈尔滨工业大学）；Peking University（北京大学）；Central South University（中南大学）；Langboat Inc.（舟博科技公司）
AstroMLab 2: AstroLLaMA-2-70B Model and Benchmarking Specialised LLMs for Astronomy
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Pennsylvania（宾夕法尼亚大学）；Indian Institute of Technology Patna（印度理工学院巴特那分校）；Center for Astrophysics Harvard University（哈佛-史密松天体物理中心）；Oak Ridge National Laboratory（橡树岭国家实验室）；The Ohio State University（俄亥俄州立大学）
Pre-trained language models (LM) such as BERT, DistilBERT, and RoBERTa can be tuned for different domains (domain-tuning) by continuing the pre-training phase on a new target domain corpus. This simple domain tuning (SDT) technique has been widely used to create domain-tuned models such as BioBERT, SciBERT and ClinicalBERT. However, during the pretraining phase on the target domain, the LM models may catastrophically forget the patterns learned from their source domain. In this work, we study the effects of catastrophic forgetting on domain-tuned LM models and investigate methods that mitigate…
