Unlocking the Potential of Model Merging for Low-Resource Languages
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);State Key Laboratory of General Artificial Intelligence, Peking University(北京大学通用人工智能国家重点实验室);Center for Data Science, Peking University(北京大学数据科学中心);College of Engineering, Peking University(北京大学工学院)
Adapting large language models (LLMs) to new languages typically involves continual pre-training (CT) followed by supervised fine-tuning (SFT). However, this CT-then-SFT approach struggles with limited data in the context of low-resource languages, failing to balance language modeling and task-solving capabilities. We thus propose model merging as an alternative for low-resource languages, combining models with distinct capabilities into a single model without additional training. We use model merging to develop task-solving LLMs for low-resource languages without SFT data in the target languages. Our experiments based on Llama-2-7B demonstrate that model merging effectively endows LLMs for low-resource languages with task-solving abilities, outperforming CT-then-SFT in scenarios with extremely scarce data. Observing performance saturation in model merging with more training tokens, we further analyze the merging process and introduce a slack variable to the model merging algorithm to mitigate the loss of important parameters, thereby enhancing performance. We hope that model merging can benefit more human languages suffering from data scarcity with its higher data efficiency.
Think Carefully and Check Again! Meta-Generation Unlocking LLMs for Low-Resource Cross-Lingual Summarization
University of California, San Diego(加利福尼亚大学圣迭戈分校);Nanyang Technological University(南洋理工大学);National University of Singapore(新加坡国立大学);Tsinghua University(清华大学)
Unlocking LLM Safeguards for Low-Resource Languages via Reasoning and Alignment with Minimal Training Data
Guangdong University of Foreign Studies(广东外语外贸大学);Guangzhou Key Laboratory of Multilingual Intelligent Processing(广州多语智能处理重点实验室);University of Pittsburgh(匹兹堡大学)
ANALOGYKB: Unlocking Analogical Reasoning of Language Models with A Million-scale Knowledge Base
School of Data Science, Fudan University(复旦大学数据科学学院);Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University(复旦大学计算机科学学院上海市数据科学重点实验室);ByteDance AI Lab(字节跳动人工智能实验室);East China Normal University(华东师范大学)
Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging
Harbin Institute of Technology(哈尔滨工业大学);University of Science and Technology of China(中国科学技术大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Shandong Agricultural University(山东农业大学);Tencent Inc.(腾讯公司);Anhui University(安徽大学)
Countering Catastrophic Forgetting of Large Language Models for Better Instruction Following via Weight-Space Model Merging
University of Florida(佛罗里达大学);Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida(佛罗里达大学医学院健康结果与生物医学信息学系);Department of Engineering Education, Herbert Wertheim College of Engineering, University of Florida(佛罗里达大学赫伯特·韦特海姆工程学院工程教育系);Preston A. Wells, Jr. Center for Brain Tumor Therapy, Lillian S. Wells Department of Neurosurgery, University of Florida(佛罗里达大学莉莉安·S·威尔斯神经外科系普雷斯顿·A·威尔斯脑肿瘤治疗中心)
Infi-MMR: Curriculum-based Unlocking Multimodal Reasoning via Phased Reinforcement Learning in Multimodal Small Language Models
The Hong Kong Polytechnic University(香港理工大学);Zhejiang University(浙江大学);University of Electronic Science and Technology of China(电子科技大学);Reallm Labs(Reallm 实验室);Amazon(亚马逊);The Hong Kong University of Science and Technology(香港理工大学)
Direct Behavior Optimization: Unlocking the Potential of Lightweight LLMs
Zhejiang Gongshang University(浙江工商大学);Zhejiang University(浙江大学);Binjiang Institute of Zhejiang University(浙江大学滨江研究院);Harbin Institute of Technology(哈尔滨工业大学)
ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMs
Shanghai University of Finance and Economics(上海财经大学);Southern University of Science and Technology(南方科技大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));The Chinese University of Hong Kong(香港中文大学)
MergeRepair: An Exploratory Study on Merging Task-Specific Adapters in Code LLMs for Automated Program Repair
University of British Columbia, Department of Computer Science(不列颠哥伦比亚大学计算机科学系);École de technologie supérieure, Department of Software and IT Engineering(高等技术学院软件与信息技术工程系)
Dyna-bAbI: unlocking bAbI's potential with dynamic synthetic benchmarking
The Hebrew University of Jerusalem(耶路撒冷希伯来大学);Allen Institute for AI(艾伦人工智能研究院);Bar-Ilan University(巴伊兰大学);Tel-Aviv University(特拉维夫大学);Stanford University(斯坦福大学)
PubTator 3.0: an AI-powered Literature Resource for Unlocking Biomedical Knowledge
National Center for Biotechnology Information (NCBI), National Library of Medicine (NLM), National Institutes of Health (NIH)(美国国立卫生研究院国家医学图书馆国家生物技术信息中心);The Neuro (Montreal Neurological Institute-Hospital), McGill University(麦吉尔大学蒙特利尔神经学研究所-医院);McGill University(麦吉尔大学);Dalian University of Technology(大连理工大学);Yale School of Medicine(耶鲁大学医学院)
本文介绍 AI 驱动的生物医学文献资源 PubTator 3.0,通过十亿级实体与关系标注、语义/关系搜索及 API 集成提升文献检索和大模型回答的可靠性。
Bridge-Coder: Unlocking LLMs' Potential to Overcome Language Gaps in Low-Resource Code
The Hong Kong University of Science and Technology(香港科技大学);Southern University of Science and Technology(南方科技大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Unconstrained Model Merging for Enhanced LLM Reasoning
The Hong Kong Polytechnic University(香港理工大学);Zhejiang University(浙江大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Reallm Labs(Reallm实验室);South China University of Technology(华南理工大学);Simon Fraser University(西蒙菲莎大学)
DeltaMerge-LowRes: Composing Language and Task Deltas for Low-Resource Adaptation
RMIT University(皇家墨尔本理工大学);Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology (HCMUT), VNU-HCM(胡志明市理工大学计算机科学与工程学院,越南胡志明市国家大学)
AP-BMM: Approximating Capability-Cost Pareto Sets of LLMs via Asynchronous Prior-Guided Bayesian Model Merging
Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies(广东新型安全智能技术重点实验室);Institute of Cyberspace Security(网络空间安全研究院);School of Computer Science and Technology(计算机科学与技术学院)