Recently introduced language model prompting methods can achieve high accuracy in zero- and few-shot settings while requiring few to no learned task-specific parameters. Nevertheless, these methods still often trail behind full model finetuning. In this work, we investigate if a dedicated continued pretraining stage could improve "promptability", i.e., zero-shot performance with natural language prompts or few-shot performance with prompt tuning. We reveal settings where existing continued pretraining methods lack promptability. We also identify current methodological gaps, which we fill with thorough large-scale experiments. We demonstrate that a simple recipe, continued pretraining that incorporates a trainable prompt during multi-task learning, leads to improved promptability in both zero- and few-shot settings compared to existing methods, up to 31% relative. On the other hand, we find that continued pretraining using MAML-style meta-learning, a method that directly optimizes few-shot promptability, yields subpar performance. We validate our findings with two prompt tuning methods, and, based on our results, we provide concrete recommendations to optimize promptability for different use cases.
Continued Pretraining of FinBERT on Finnish Histopathological Reports: Train-Time Signals and Proxy Downstream Correlations
Digital Workforce Services（数字劳动力服务）；Heart and Lung Center, Helsinki University Hospital（赫尔辛基大学医院心脏和肺中心）；Central Finland Biobank（中央芬兰生物库）；Wellbeing Services County of Central Finland（中央芬兰县福祉服务）
OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining
Center for Information and Language Processing, LMU Munich（慕尼黑大学信息与语言处理中心）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Bosch Center for Artificial Intelligence（博世人工智能中心）
Curió-Edu 7B: Examining Data Selection Impacts in LLM Continued Pretraining
Institute of Computing (IC) University of Campinas (UNICAMP)（计算研究所（IC）坎皮纳斯大学（UNICAMP））；School of Electrical and Computer Engineering (FEEC) University of Campinas (UNICAMP)（电气与计算机工程学院（FEEC）坎皮纳斯大学（UNICAMP））
Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting
Concordia University（康考迪亚大学）；Mila – Quebec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；Laval University（拉瓦尔大学）；University of Tübingen（蒂宾根大学）；Tübingen AI Center（蒂宾根人工智能中心）
Beyond Captions: Context-Grounded Reconstruction for Biomedical Multimodal Continued Pretraining
The Hong Kong Polytechnic University（香港理工大学）；Sun Yat-sen University（中山大学）；InfiX.ai；PolyU-Daya Bay Technology and Innovation Research Institute（PolyU-大亚湾技术与创新研究院）
EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training
Institute of Computer Science, University of Tartu（塔尔图大学计算机科学研究院）；Department of Software Science, Tallinn University of Technology（塔林技术大学软件科学系）；Institute of the Estonian Language, Tallinn, Estonia（爱沙尼亚语言研究院）；School of Humanities, Tallinn University（塔林大学人文学院）
AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages
University of Colorado Boulder（科罗拉多大学博尔德分校）；University of Stuttgart（斯图加特大学）；University of Edinburgh（爱丁堡大学）；Microsoft Turing（微软图灵）；Facebook AI Research（脸书人工智能研究院）；New York University（纽约大学）；Universidad de la República, Uruguay（乌拉圭共和国大学）；Universidad Tecnológica de Tlaxcala（特拉斯卡拉理工大学）；University of Zurich（苏黎世大学）；Carnegie Mellon University（卡内基梅隆大学）；Universidad Nacional Autónoma de México（墨西哥国立自治大学）；Universidade Tecnológica Federal do Paraná（巴拉那联邦理工大学）；Dartmouth College（达特茅斯学院）
From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corpora
Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Technical University of Munich（慕尼黑技术大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Minzu University of China（民族大学）；Institute for AI, Tsinghua University（清华大学人工智能研究院）
Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Institute for Language, Cognition and Computation, University of Edinburgh（爱丁堡大学语言、认知与计算研究所）；Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）
Small or Large? Zero-Shot or Finetuned? Guiding Language Model Choice for Specialized Applications in Healthcare
British Columbia Cancer Registry Provincial Health Services Authority（不列颠哥伦比亚省癌症登记处省级卫生服务局）；Data Science Institute University of British Columbia（数据科学研究所不列颠哥伦比亚大学）
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting
Brown University（布朗大学）；EleutherAI；Yale University（耶鲁大学）；Hugging Face；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；University College London（伦敦大学学院）；PSAU（沙特王子萨塔姆·本·阿卜杜勒阿齐兹大学）；Nanyang Technological University（南洋理工大学）；University of Washington（华盛顿大学）；University of Oregon（俄勒冈大学）；The Hong Kong University of Science and Technology（香港科技大学）；Booz Allen Hamilton（博思艾伦咨询公司）；NAVER LABS Europe（NAVER欧洲实验室）
