Domain Adaptation of Llama3-70B-Instruct through Continual Pre-Training and Model Merging: A Comprehensive Evaluation
Arcee
Shamane Siriwardhana, Mark McQuade, Thomas Gauthier, Lucas Atkins, Fernando Fernandes Neto, Luke Meyers, Anneketh Vij, Tyler Odenthal, Charles Goddard, Mary Mac… 展开作者
Shamane Siriwardhana, Mark McQuade, Thomas Gauthier, Lucas Atkins, Fernando Fernandes Neto, Luke Meyers, Anneketh Vij, Tyler Odenthal, Charles Goddard, Mary MacCarthy, Jacob Solawetz
We conducted extensive experiments on domain adaptation of the Meta-Llama-3-70B-Instruct model on SEC data, exploring its performance on both general and domain-specific benchmarks. Our focus included continual pre-training (CPT) and model merging, aiming to enhance the model's domain-specific capabilities while mitigating catastrophic forgetting. Through this study, we evaluated the impact of integrating financial regulatory data into a robust language model and examined the effectiveness of our model merging techniques in preserving and improving the model's instructive abilities. The model is accessible at hugging face: https://huggingface.co/arcee-ai/Llama-3-SEC-Base, arcee-ai/Llama-3-SEC-Base. This is an intermediate checkpoint of our final model, which has seen 20B tokens so far. The full model is still in the process of training. This is a preprint technical report with thorough evaluations to understand the entire process.
MagMax: Leveraging Model Merging for Seamless Continual Learning
IDEAS NCBR；Warsaw University of Technology（华沙工业大学）；Autonomous University of Barcelona（巴塞罗那自治大学）；Computer Vision Center（计算机视觉中心）；Tooploox；Gdańsk University of Technology（格但斯克工业大学）
Modular Delta Merging with Orthogonal Constraints: A Scalable Framework for Continual and Reversible Model Composition
Rensselaer Polytechnic Institute（伦斯勒理工学院）；National University of Sciences and Technology (NUST)（国立科技大学）；Virginia Tech（弗吉尼亚理工大学）；University of New Mexico（新墨西哥大学）
Continual Instruction Tuning for Large Multimodal Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；The Peng Cheng Laboratory（鹏城实验室）；Wuhan AI Research（武汉人工智能研究院）
ChainLoRA: Geometry-Preserving Task Vector Merging for Continual Learning in LLMs
Shanghai Jiao Tong University（上海交通大学）；Shanghai Innovation Institute（上海创新研究院）；Beijing Zhongguancun Academy（北京中关村学院）；StepFun（阶跃星辰）；Huazhong University of Science and Technology（华中科技大学）；IAAR（智能自动化研究院）
Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning
School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；State Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）
When Muon Meets Task Interference: A Spectral Perspective on Continual Learning and Model Merging
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学现代国家重点实验室（注：此处按原文直译，实际常见译名为“南京大学计算机软件新技术国家重点实验室”））；University of Wollongong（卧龙岗大学）
University of Bremen（不来梅大学）；Jeonbuk National University（全北国立大学）；TU Darmstadt（达姆施塔特工业大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；Hessian Center for AI (hessian.AI)（黑森人工智能中心）；University College London（伦敦大学学院）
Unlocking the Potential of Model Merging for Low-Resource Languages
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）；Center for Data Science, Peking University（北京大学数据科学中心）；College of Engineering, Peking University（北京大学工学院）
Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities
Shenzhen Campus of Sun Yat-sen University, China（中山大学深圳校区）；Northeastern University China（东北大学）；Shenzhen Campus of Sun Yat-sen University China（中山大学深圳校区）；Nanyang Technological University Singapore（南洋理工大学）；Northeastern University（东北大学）；Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Nanyang Technological University（南洋理工大学）；Institute for Clarity in Documentation Dublin Ohio USA（文档清晰研究所）；Inria Paris-Rocquencourt Rocquencourt France（巴黎-罗quentourt研究所）；Rajiv Gandhi University Doimukh Arunachal Pradesh India（拉贾·甘地大学）；Tsinghua University Haidian Qu Beijing Shi China（清华大学）；Palmer Research Laboratories San Antonio Texas USA（帕勒研究中心）；Institute for Clarity in Documentation（文档清晰研究所）；Inria Paris-Rocquencourt（巴黎-罗quentourt研究所）；Rajiv Gandhi University（拉贾·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕勒研究中心）
Continual-NExT: A Unified Comprehension And Generation Continual Learning Framework
East China Normal University（华东师范大学）；Shanghai Innovation Institute（上海创新研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Xiamen University（厦门大学）
East China Normal University（东华大学）；Shanghai Innovation Institute（上海创新研究院）；Shanghai AI Laboratory（上海人工智能实验室）；Xiamen University（厦门大学）；Tencent YouTu Lab（腾讯YouTu实验室）
SwitchCIT: Switching for Continual Instruction Tuning
Coordinated Science Laboratory（协调科学实验室）；Department of Electrical and Computer Engineering（电气与计算机工程系）；Department of Mathematics（数学系）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models
Institute of Intelligent Information Processing, Shanxi University（山西大学智能信息处理研究所）；Alliance Manchester Business School, The University of Manchester（曼彻斯特大学联盟曼彻斯特商学院）
