Flexible and Adaptable Summarization via Expertise Separation
KAUST(阿卜杜拉国王科技大学);MBZUAI(Mohamed bin Zayed 人工智能大学);Ant Group(蚂蚁集团);Shandong University(山东大学);Peking University(北京大学);Renmin University of China(中国人民大学);University of Notre Dame(圣母大学)
A proficient summarization model should exhibit both flexibility -- the capacity to handle a range of in-domain summarization tasks, and adaptability -- the competence to acquire new knowledge and adjust to unseen out-of-domain tasks. Unlike large language models (LLMs) that achieve this through parameter scaling, we propose a more parameter-efficient approach in this study. Our motivation rests on the principle that the general summarization ability to capture salient information can be shared across different tasks, while the domain-specific summarization abilities need to be distinct and tailored. Concretely, we propose MoeSumm, a Mixture-of-Expert Summarization architecture, which utilizes a main expert for gaining the general summarization capability and deputy experts that selectively collaborate to meet specific summarization task requirements. We further propose a max-margin loss to stimulate the separation of these abilities. Our model's distinct separation of general and domain-specific summarization abilities grants it with notable flexibility and adaptability, all while maintaining parameter efficiency. MoeSumm achieves flexibility by managing summarization across multiple domains with a single model, utilizing a shared main expert and selected deputy experts. It exhibits adaptability by tailoring deputy experts to cater to out-of-domain few-shot and zero-shot scenarios. Experimental results on 11 datasets show the superiority of our model compared with recent baselines and LLMs. We also provide statistical and visual evidence of the distinct separation of the two abilities in MoeSumm (https://github.com/iriscxy/MoE_Summ).
Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Survey
Mohamed bin Zayed University of Artificial Intelligence(Mohamed bin Zayed大学);Northeastern University(东北大学);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);ByteDance(字节跳动);Wuhan University(武汉大学)
Countering Catastrophic Forgetting of Large Language Models for Better Instruction Following via Weight-Space Model Merging
University of Florida(佛罗里达大学);Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida(佛罗里达大学医学院健康结果与生物医学信息学系);Department of Engineering Education, Herbert Wertheim College of Engineering, University of Florida(佛罗里达大学赫伯特·韦特海姆工程学院工程教育系);Preston A. Wells, Jr. Center for Brain Tumor Therapy, Lillian S. Wells Department of Neurosurgery, University of Florida(佛罗里达大学莉莉安·S·威尔斯神经外科系普雷斯顿·A·威尔斯脑肿瘤治疗中心)
One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support
University of Bern(伯尔尼大学);University of Copenhagen(哥本哈根大学);Bern University of Applied Sciences(伯尔尼应用科学大学);University of Fribourg(弗里堡大学);Stanford University(斯坦福大学)