Language models (LMs) are trained on web text originating from many points in time and, in general, without any explicit temporal grounding. This work investigates the temporal chaos of pretrained LMs and explores various methods to align their internal knowledge to a target time, which we call "temporal alignment." To do this, we first automatically construct a dataset containing 20K time-sensitive questions and their answers for each year from 2000 to 2023. Based on this dataset, we empirically show that pretrained LMs (e.g., LLaMa2), despite having a recent pretraining cutoff (e.g., 2022), mostly answer questions using earlier knowledge (e.g., in 2019). We then develop several methods, from prompting to finetuning, to align LMs to use their most recent knowledge when answering questions, and investigate various factors in this alignment. Our experiments demonstrate that aligning LLaMa2 to the year 2022 can enhance its performance by up to 62% according to that year's answers. This improvement occurs even without explicitly mentioning time information, indicating the possibility of aligning models' internal sense of time after pretraining. Finally, we find that alignment to a historical time is also possible, with up to 2.8$\times$ the performance of the unaligned LM in 2010 if finetuning models to that year. These findings hint at the sophistication of LMs' internal knowledge organization and the necessity of tuning them properly.
Few-shot Knowledge Graph-to-Text Generation with Pretrained Language Models
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Beijing Key Laboratory of Big Data Management and Analysis Methods（大数据管理与分析方法北京市重点实验室）；Huawei Cloud（华为云）；School of Information, Renmin University of China（中国人民大学信息学院）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）
KBioXLM: A Knowledge-anchored Biomedical Multilingual Pretrained Language Model
Institute of Artificial Intelligence, Soochow University（苏州大学人工智能研究院）；Peking University（北京大学）；The Hong Kong Polytechnic University（香港理工大学）；Pharmcube（药智魔方）；Changping Laboratory（昌平实验室）
Harnessing Structural Context for Entity Alignment Foundation Models
State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China（南京大学新型软件技术国家重点实验室）；Nanjing University of Information Science and Technology, Nanjing, China（南京信息科学技术大学）；National Institute of Healthcare Data Science, Nanjing University, Nanjing, China（南京大学健康数据科学国家研究院）
Meeting the Needs of Low-Resource Languages: The Value of Automatic Alignments via Pretrained Models
University of Colorado Boulder（科罗拉多大学博尔德分校）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of Edinburgh（爱丁堡大学）；Universidad de la República（共和国大学）；Northeastern University（东北大学）；Universidade Tecnológica Federal do Paraná（巴拉那联邦技术大学）；Dartmouth College（达特茅斯学院）
ITA: Image-Text Alignments for Multi-Modal Named Entity Recognition
Shopee（虾皮）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉成像工程技术研究中心）；Microsoft（微软）；Alibaba Group（阿里巴巴集团）
Speak in Context: Multilingual ASR with Speech Context Alignment via Contrastive Learning
Institute for Analytics and Data Science, University of Essex（数据分析与科学研究所，埃塞克斯大学）；School of Computer Science and Electronic Engineering, University of Essex（计算机科学与电子工程学院，埃塞克斯大学）
UBERT: A Novel Language Model for Synonymy Prediction at Scale in the UMLS Metathesaurus
AI Institute, University of South Carolina（南卡罗来纳大学人工智能研究所）；National Library of Medicine（美国国家医学图书馆）；The Ohio State University（俄亥俄州立大学）；George Washington University（乔治华盛顿大学）
Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
Department of Computer Science and Engineering, The Ohio State University（计算机科学与工程系，俄亥俄州立大学）；Translational Data Analytics Institute, The Ohio State University（转化数据分析研究所，俄亥俄州立大学）；Department of Biomedical Informatics, The Ohio State University（生物医学信息学系，俄亥俄州立大学）
Temporal Order Preserved Optimal Transport-based Cross-modal Knowledge Transfer Learning for ASR
National Institute of Information and Communications Technology（信息通信研究机构）；Research Center for Information Technology Innovation（资讯科技创新研究中心）；Academia Sinica（中央研究院）
