上下文学习是指预训练语言模型从其上下文中的任务示例和指令中学习执行任务,在NLP社区引起了广泛关注。然而,上下文学习的能力并未被充分利用,因为语言模型没有被显式训练以在上下文中学习。为此,我们提出PICL(Pre-training for In-Context Learning),这是一个通过使用简单的语言建模目标,在通用纯文本语料库的大量“内在任务”上预训练模型,从而增强语言模型上下文学习能力的框架。PICL鼓励模型通过以上下文为条件来推断和执行任务,同时保持预训练模型的任务泛化能力。我们在七个广泛使用的文本分类数据集和Super-NaturalInstrutions基准上评估了使用PICL训练的模型的上下文学习性能,该基准包含100多个被表述为文本生成的NLP任务。我们的实验表明,PICL比一系列基线更有效且更具任务泛化性,优于参数量约为4倍的更大语言模型。代码公开于https://github.com/thu-coai/PICL。
英文摘要:
In-context learning, where pre-trained language models learn to perform tasks from task examples and instructions in their contexts, has attracted much attention in the NLP community. However, the ability of in-context learning is not fully exploited because language models are not explicitly trained to learn in context. To this end, we propose PICL (Pre-training for In-Context Learning), a framework to enhance the language models' in-context learning ability by pre-training the model on a large collection of "intrinsic tasks" in the general plain-text corpus using the simple language modeling objective. PICL encourages the model to infer and perform tasks by conditioning on the contexts while maintaining task generalization of pre-trained models. We evaluate the in-context learning performance of the model trained with PICL on seven widely-used text classification datasets and the Super-NaturalInstrctions benchmark, which contains 100+ NLP tasks formulated to text generation. Our experiments show that PICL is more effective and task-generalizable than a range of baselines, outperforming larger language models with nearly 4x parameters. The code is publicly available at https://github.com/thu-coai/PICL.
LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Zhongguancun Academy, Beijing, China(中关村学院,北京,中国);Wuhan AI Research(武汉人工智能研究所)
Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contexts
Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China(教育部多媒体可信感知与高效计算重点实验室);Xiamen University(厦门大学);Peking University(北京大学);University of Illinois Urbana-Champaign(伊利诺伊大学香槟分校);Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室)
Training-free Context-adaptive Attention for Efficient Long Context Modeling
School of Future Technology, South China University of Technology, Guangzhou, China(未来技术学院,华南理工大学,广州,中国);Pengcheng Laboratory, Shenzhen, China(鹏城实验室,深圳,中国);School of Software Engineering, South China University of Technology, Guangzhou, China(软件学院,华南理工大学,广州,中国);Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学,深圳,中国)
LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models
National University of Singapore(新加坡国立大学);Nanjing University(南京大学);University of Toronto(多伦多大学);Mila, Québec AI Institute(魁北克人工智能研究所(Mila));Université de Montréal(蒙特利尔大学);Tencent Inc(腾讯公司);Baidu Inc(百度公司);Nanyang Technological University(南洋理工大学)
Training-Inference Consistent Segmented Execution for Long-Context LLMs
College of Computer Science, Inner Mongolia University, Hohhot 010021, China;National \& Local Joint Engineering Research Center of Intelligent Information Processing Technology for Mongolian, Hohhot 010021, China;Inner Mongolia Key Laboratory of Multilingual Artificial Intelligence Technology, Hohhot 010021, China;Thrust of Artificial Intelligence, The Hong Kong University of Science
E^2-LLM: Efficient and Extreme Length Extension of Large Language Models
Alibaba Group(阿里巴巴集团);University of Waterloo(滑铁卢大学);The Chinese University of Hong Kong(香港中文大学);The Hong Kong University of Science and Technology(香港科技大学)
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
Microsoft Corporation(微软公司);School of Computer Science, Peking University(北京大学计算机学院);National Key Laboratory for Multimedia Information Processing, Peking University(北京大学多媒体信息处理国家重点实验室)