Despite the remarkable capabilities of Language Models (LMs) across diverse tasks, no single model consistently outperforms others, necessitating efficient methods to combine their strengths without expensive retraining. Existing model merging techniques, such as parameter averaging and task-guided fusion, often rely on data-dependent computations or fail to preserve internal knowledge, limiting their robustness and scalability. We introduce SeMe (Semantic-based Merging), a novel, data-free, and training-free approach that leverages latent semantic alignment to merge LMs at a fine-grained, layer-wise level. Unlike prior work, SeMe not only preserves model behaviors but also explicitly stabilizes internal knowledge, addressing a critical gap in LM fusion. Through extensive experiments across diverse architectures and tasks, we demonstrate that SeMe outperforms existing methods in both performance and efficiency while eliminating reliance on external data. Our work establishes a new paradigm for knowledge-aware model merging and provides insights into the semantic structure of LMs, paving the way for more scalable and interpretable model composition.
Edinburgh Clinical NLP at SemEval-2024 Task 2: Fine-tune your model unless you have access to GPT-4
School of Informatics, University of Edinburgh(爱丁堡大学信息学院);Usher Institute, University of Edinburgh(爱丁堡大学厄舍研究所);Edinburgh Futures Institute, University of Edinburgh(爱丁堡未来学院);School of Literatures, Languages and Cultures, University of Edinburgh(爱丁堡大学文学、语言与文化学院)
本文介绍了SemEval-2020任务1中使用的一种无监督检测词汇语义变化的方法SChME(Semantic Change Detection with Model Enclusion)。SChME使用一个模型集成结合信号的分布模型(词嵌入)和词频模型,其中每个模型投下一票,表明一个词的概率sufficientsemantic变化根据该功能。更具体地说,我们将词向量的余弦距离与我们命名为映射邻域距离(MAP)的基于邻域的度量相结合,并将词频差分度量作为我们模型的输入信号。此外,我们还探索了基于标记的方法来研究在此过程中使用的地标的重要性。我们的研究结果表明,用于对齐的地标的数量对模型的预测性能有直接的影响。此外,我们表明,语言,sufferless语义变化往往受益于使用大量的地标,而languageswith更多的语义变化受益于一个更谨慎的选择地标数对齐。
Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging
Harbin Institute of Technology(哈尔滨工业大学);University of Science and Technology of China(中国科学技术大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Shandong Agricultural University(山东农业大学);Tencent Inc.(腾讯公司);Anhui University(安徽大学)
A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignment
Microsoft Healthcare & Life Sciences(微软医疗与生命科学);Microsoft Research Montréal(微软研究院蒙特利尔);Mila, Université de Montréal(蒙特利尔大学米拉研究所);IKIM, University Hospital Essen(埃森大学医院人工智能医学研究所);Cancer Research Center Cologne Essen(科隆埃森癌症研究中心);German Cancer Consortium(德国癌症联盟);TU Dortmund(多特蒙德工业大学)
Multi-Objective Alignment of Language Models for Personalized Psychotherapy
Department of Computer Science, University of California, Los Angeles, CA, USA(计算机科学系,加州大学洛杉矶分校);Department of Psychology, The University of Texas at Austin, Austin, TX, USA(心理学系,德克萨斯大学奥斯汀分校);Department of Psychiatry, NYU Grossman School of Medicine, New York, NY, USA(精神病学系,纽约大学格罗斯曼医学院)
SEG:Seeds-Enhanced Iterative Refinement Graph Neural Network for Entity Alignment
College of Computer;Mathematics Central South University of Forestry;4em College of Computer;Mathematics 4em Central South University of Forestry;3em Department of Computer Science 3em State University of New York 3em New Paltz, New York 12561, USA 4em