We introduce InfiFusion, an efficient training pipeline designed to integrate multiple domain-specialized Large Language Models (LLMs) into a single pivot model, effectively harnessing the strengths of each source model. Traditional fusion methods either merge model parameters directly or rely on knowledge distillation with rigid assumptions, limiting their flexibility and efficiency. InfiFusion overcomes these limitations by enhancing Universal Logit Distillation (ULD) with Top-K selection and Logits Standardization. We propose two fusion strategies: Pairwise Fusion (InfiFusion$_p$), where each source model knowledge is distilled individually into the pivot model followed by merging and Unified Fusion (InfiFusion$_u$), where knowledge from all source models is distilled simultaneously into the pivot model. InfiFusion outperforms the state-of-the-art models, such as Qwen-2.5-14B-Instruct and Phi-4, across 11 widely applied benchmarks covering reasoning, coding, mathematics, and instruction-following tasks. Notably, InfiFusion achieves this superior performance while significantly reduces computational costs, completing full training with only 160 H800 GPU hours compared to the millions typically required for traditional LLM training.
MathFusion: Enhancing Mathematical Problem-solving of LLM through Instruction Fusion
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学北京校区人工智能学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Tsinghua University(清华大学);Shanghai Jiao Tong University(上海交通大学);Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE(教育部下一代智能搜索与推荐工程技术研究中心);School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院)
Triadic Fusion of Cognitive, Functional, and Causal Dimensions for Explainable LLMs: The TAXAL Framework
Department of Computer Science and Artificial Intelligence(计算机科学与人工智能系);Andalusian Institute of Data Science and Computational Intelligence(安达卢西亚数据科学与计算智能研究所);University of Granada(格拉纳达大学)
A Multi-granularity Concept Sparse Activation and Hierarchical Knowledge Graph Fusion Framework for Rare Disease Diagnosis
School of Software, Yunnan University(云南大学软件学院);Yunnan Key Laboratory of Software Engineering(云南软件工程重点实验室);Yunnan Provincial Hospital of Traditional Chinese Medicine(云南中医药省人民医院)
FASIONAD++ : Integrating High-Level Instruction and Information Bottleneck in FAt-Slow fusION Systems for Enhanced Safety in Autonomous Driving with Adaptive Feedback
The School of Vehicle and Mobility, Tsinghua University(清华大学车辆与出行学院);McGill University(麦吉尔大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Waseda University(早稻田大学);University of Minnesota(明尼苏达大学);AI2Robotics
MedAide: Information Fusion and Anatomy of Medical Intents via LLM-based Agent Collaboration
1 College of Intelligent Robotics;Advanced Manufacturing, Fudan University \, 2 Multimodal Foundation Research Team, CIT Lab [1mm] 3 Shanghai Artificial Intelligence Laboratory \, 4 Institute of Science;Technology for Brain-Inspired Intelligence, Fudan University [1mm] 5 School of Computing;Mathematical Sciences, University of Leicester \, 6 Cardiff University [1mm] 7 Department of Computer Science, The University of Toronto \, 8 Institute of Metaverse \& Intelligent Medicine [1mm] 9 Jilin Provincial Key Laboratory of Intelligence Science;Engineering [1mm] 10 Engineering Research Center of AI
InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models
The Hong Kong Polytechnic University (PolyU)(香港理工大学);Zhejiang University(浙江大学);PolyU-Daya Bay Technology and Innovation Research Institute(香港理工大学-大亚湾技术与创新研究院)
Multimodal Fusion with LLMs for Engagement Prediction in Natural Conversation
Computer Science Department, Carnegie Mellon University(卡内基梅隆大学计算机科学系);Robotics Institute, Carnegie Mellon University(卡内基梅隆大学机器人研究所);Neuroscience Institute, Carnegie Mellon University(卡内基梅隆大学神经科学研究所);Department of Psychology, The University of Texas at Austin(德克萨斯大学奥斯汀分校心理学系);Center for Perceptual Systems, The University of Texas at Austin(德克萨斯大学奥斯汀分校感知系统中心)
Temporal-Aware Heterogeneous Graph Reasoning with Multi-View Fusion for Temporal Question Answering
1 School of Internet of Things, Nanjing University of Posts;Telecommunications 2 Faculty of Medicine \& Health, University of New South Wales, Sydney, Australia 3 Dept. of Biomedical Engineering \& Biotechnology, Khalifa University, Abu Dhabi 4 Beijing Information Science;Technology University 5 Shortest Path Technology Co., Ltd.
LoMo: Local Modality Substitution for Deeper Vision-Language Fusion
Fudan University(复旦大学);Shanghai Innovation Institute(上海创新研究院);Shanghai Jiao Tong University(上海交通大学);University of Science and Technology of China(中国科学技术大学)
TCAR-Gen: Temporal Graph Retrieval with Evidence Fusion for Knowledge-Grounded Generation
Dipartimento di Informatica, Università di Verona(威尼斯大学计算机科学系);School of Advanced Studies, University of Camerino(坎皮诺大学高级研究学院);Department of Computer Science, School of Mathematics and Computer Science, Institute of Business Administration (IBA), Karachi, Pakistan(卡拉奇工商管理学院(IBA)数学与计算机科学学院计算机科学系)