The University of Hong Kong（香港大学）；ARC Lab, Tencent PCG（腾讯平台与内容事业群ARC实验室）；Shanghai Jiao Tong University（上海交通大学）；Beijing Language and Culture University（北京语言大学）
Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ying Shan, Ping Luo
Humans generally acquire new skills without compromising the old; however, the opposite holds for Large Language Models (LLMs), e.g., from LLaMA to CodeLLaMA. To this end, we propose a new post-pretraining method for LLMs with an expansion of Transformer blocks. We tune the expanded blocks using only new corpus, efficiently and effectively improving the model's knowledge without catastrophic forgetting. In this paper, we experiment on the corpus of code and math, yielding LLaMA Pro-8.3B, a versatile foundation model initialized from LLaMA2-7B, excelling in general tasks, programming, and mathematics. LLaMA Pro and its instruction-following counterpart (LLaMA Pro-Instruct) achieve advanced performance among various benchmarks, demonstrating superiority over existing open models in the LLaMA family and the immense potential of reasoning and addressing diverse tasks as an intelligent agent. Our findings provide valuable insights into integrating natural and programming languages, laying a solid foundation for developing advanced language agents that operate effectively in various environments.
DSPC: Dual-Stage Progressive Compression Framework for Efficient Long-Context Reasoning
Institute of Cyberspace Security, Zhejiang University of Technology A STAR Amazon Binjiang Institute of Artificial Intelligence, Zhejiang University of Technology
Dynamic data sampler for cross-language transfer learning in large language models
School of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；Tencent AI Lab（腾讯AI实验室）；LIESMARS, Wuhan University（武汉大学遥感信息工程学院（国际数字地球学会实验室））；School of AI and Advanced Computing, Xi’an Jiaotong-Liverpool（西交利物浦大学人工智能与先进计算学院）
TableDreamer: Progressive and Weakness-guided Data Synthesis from Scratch for Table Instruction Tuning
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Baidu Inc（百度公司）
LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
Soochow University（苏州大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）
SOCIALITE-LLAMA: An Instruction-Tuned Model for Social Scientific Tasks
Stony Brook University（石溪大学）；JP Morgan Chase（摩根大通）；Bloomberg（彭博公司）；Preferabli（普雷费布利公司）；University of Pennsylvania（宾夕法尼亚大学）；National Institute of Health（美国国立卫生研究院）；Grammarly（grammarly公司）
Radiology-Llama2: Best-in-Class Large Language Model for Radiology
University of Georgia（佐治亚大学）；Harvard University（哈佛大学）；Second Xiangya Hospital（中南大学湘雅二医院）；Northwestern Polytechnical University（西北工业大学）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Mayo Clinic（梅奥诊所）；ShanghaiTech University（上海科技大学）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能医疗科技有限公司）；Shanghai Clinical Research and Trial Center（上海临床研究中心）
Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval
Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；University of Science and Technology of China（中国科学技术大学）；The Hong Kong Polytechnic University（香港理工大学）
Ophtha-LLaMA2: A Large Language Model for Ophthalmology
University of Electronic Science and Technology of China（电子科技大学）；The First Affiliated Hospital of Nanchang University（南昌大学第一附属医院）；Jiangxi Medical College（江西医学院）；Eye & ENT Hospital of Fudan University（复旦大学附属眼耳鼻喉科医院）；University of Georgia（佐治亚大学）；Northwestern Polytechnical University（西北工业大学）
Hong Kong Metropolitan University（香港都会大学）；Hong Kong Polytechnic University（香港理工大学）；Shanghai 100me Internet Technology Co., Ltd.（上海一零零米互联网科技有限公司）；Lingnan University（岭南大学）
Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a survey
University of Kurdistan（库尔德斯坦大学）；Queen Mary University（女王玛丽大学）；Torrens University Australia（澳大利亚托伦斯大学）；University of Technology Sydney（悉尼技术大学）；University of Kurdistan Sanandaj, Iran（伊朗桑和杰德大学）；TU Dortmund University（多特蒙德技术大学）；University of Kurdistan Hewler（库尔德斯坦大学海勒）；Tehran University（德黑兰大学）
MGH Radiology Llama: A Llama 3 70B Model for Radiology
School of Computing, University of Georgia（佐治亚大学计算学院）；Department of Radiology, Massachusetts General Hospital and Harvard Medical School（麻省总医院及哈佛医学院放射科）
DRG-LLaMA : Tuning LLaMA Model to Predict Diagnosis-related Group for Hospitalized Patients
Mayo Clinic Health System（妙佑医疗国际医疗系统）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Mayo Clinic（妙佑医疗国际）；Carle Illinois College of Medicine（卡勒伊利诺伊医学院）
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）
