Large language models (LLMs) demonstrate strong task-specific capabilities through fine-tuning, but merging multiple fine-tuned models often leads to degraded performance due to overlapping instruction-following components. Task Arithmetic (TA), which combines task vectors derived from fine-tuning, enables multi-task learning and task forgetting but struggles to isolate task-specific knowledge from general instruction-following behavior. To address this, we propose Layer-Aware Task Arithmetic (LATA), a novel approach that assigns layer-specific weights to task vectors based on their alignment with instruction-following or task-specific components. By amplifying task-relevant layers and attenuating instruction-following layers, LATA improves task learning and forgetting performance while preserving overall model utility. Experiments on multiple benchmarks, including WikiText-2, GSM8K, and HumanEval, demonstrate that LATA outperforms existing methods in both multi-task learning and selective task forgetting, achieving higher task accuracy and alignment with minimal degradation in output quality. Our findings highlight the importance of layer-wise analysis in disentangling task-specific and general-purpose knowledge, offering a robust framework for efficient model merging and editing.
Disentangling Mathematical Reasoning in LLMs: A Methodological Investigation of Internal Mechanisms
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Saarland University（萨尔兰大学）；Center for European Research in Trusted AI (CERTAIN)（可信人工智能欧洲研究中心）
Disentangling Language Roles in Multilingual LLM Task Execution
Marquette（马凯特大学）；Cornell（康奈尔大学）；UC San Diego（南加州大学圣地亚哥分校）；UPenn（普林斯顿大学）；UT Austin（德克萨斯大学奥斯汀分校）；Maryland（马里兰大学）；Michigan（密歇根大学）；UIUC（伊利诺伊大学香槟分校）；Melbourne（墨尔本大学）；Stanford（斯坦福大学）
Synthesizing Instruction-Tuning Datasets with Contrastive Decoding
Department of Computer Science, School of Computing, Institute of Science Tokyo（东京科学大学计算机科学系）；National Institute of Advanced Industrial Science and Technology（国家先进工业科学与技术研究院）；Research and Development Center for Large Language Models, NII（大型语言模型研究与开发中心，日本信息产业技术综合研究所）
Wait, that's not an option: LLMs Robustness with Incorrect Multiple-Choice Options
University of Warsaw（华沙大学）；Institute of Mathematics, Polish Academy of Sciences（波兰科学院数学研究所）；MIM Solutions（MIM解决方案）；K-Scale Labs（K-Scale实验室）；IDEAS NCBR；IDEAS Research Institute（IDEAS研究学院）
