NeKo: Cross-Modality Post-Recognition Error Correction with Tasks-Guided Mixture-of-Experts Language Model
NVIDIA
Yen-Ting Lin, Zhehuai Chen, Piotr Zelasko, Zhen Wan, Xuesong Yang, Zih-Ching Chen, Krishna C Puvvada, Szu-Wei Fu, Ke Hu, Jun Wei Chiu, Jagadeesh Balam, Boris Gi… 展开作者
Yen-Ting Lin, Zhehuai Chen, Piotr Zelasko, Zhen Wan, Xuesong Yang, Zih-Ching Chen, Krishna C Puvvada, Szu-Wei Fu, Ke Hu, Jun Wei Chiu, Jagadeesh Balam, Boris Ginsburg, Yu-Chiang Frank Wang, Chao-Han Huck Yang
构建一个通用的后识别纠错器面临关键问题:如何在大量领域数据集的混合中最有效地训练模型?答案在于学习领域特定特征并在单个模型中消化其知识。以往的方法通过使用单独的纠错语言模型来实现这一点,导致参数显著增加。在本工作中,我们提出专家混合(MoE)作为解决方案,强调MoE远不止是一种可扩展性工具。我们提出了一种多任务纠错MoE,其中训练专家成为语音到文本、语言到文本和视觉到文本数据集的“专家”,通过学习将每个数据集的标记路由到其映射的专家。在Open ASR Leaderboard上的实验表明,我们通过实现平均相对5.0%的WER减少和语音和翻译任务中BLEU分数的显著提升,探索了新的最先进的性能。在零样本评估中,NeKo在Hyporadise基准测试中以15.5%到27.6%的相对WER减少优于GPT-3.5和Claude-Opus。NeKo在语法和后OCR纠错任务中作为多任务模型表现出竞争力。
英文摘要:
Construction of a general-purpose post-recognition error corrector poses a crucial question: how can we most effectively train a model on a large mixture of domain datasets? The answer would lie in learning dataset-specific features and digesting their knowledge in a single model. Previous methods achieve this by having separate correction language models, resulting in a significant increase in parameters. In this work, we present Mixture-of-Experts as a solution, highlighting that MoEs are much more than a scalability tool. We propose a Multi-Task Correction MoE, where we train the experts to become an ``expert'' of speech-to-text, language-to-text and vision-to-text datasets by learning to route each dataset's tokens to its mapped expert. Experiments on the Open ASR Leaderboard show that we explore a new state-of-the-art performance by achieving an average relative 5.0% WER reduction and substantial improvements in BLEU scores for speech and translation tasks. On zero-shot evaluation, NeKo outperforms GPT-3.5 and Claude-Opus with 15.5% to 27.6% relative WER reduction in the Hyporadise benchmark. NeKo performs competitively on grammar and post-OCR correction as a multi-task model.
Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);ZJU-Ant Group Joint Lab of Knowledge Graph(浙江大学-蚂蚁集团知识图谱联合实验室);Ant Group(蚂蚁集团);School of Software Technology, Zhejiang University(浙江大学软件学院);Zhejiang Key Laboratory of Big Data Intelligent Computing(浙江省大数据智能计算重点实验室)
ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
NepAl Applied Mathematics and Informatics Institute(尼泊尔应用数学与信息学研究所);Fogsphere (Redev.AI Ltd)(福格球(Redev.AI有限公司));University of Lausanne(洛桑大学);West Virginia University(西弗吉尼亚大学);University of Verona(维罗纳大学);University College London(伦敦大学学院);University of Aberdeen(阿伯丁大学)
The University of Edinburgh(爱丁堡大学);Imperial College, London(伦敦帝国学院);University of Surrey(萨里大学);Queen Mary University of London(伦敦玛丽女王大学);Microsoft Research, Cambridge(微软研究院剑桥分部)
CAME-AB: Cross-Modality Attention with Mixture-of-Experts for Antibody Binding Site Prediction
Generative AI Research and Development Center The Hong Kong University of Science and Technology(生成式人工智能研究与发展中心 香港科学大学);MILES The University of Hong Kong(MILES 香港大学);College of Veterinary Medicine Jilin University(吉林大学兽医学院);School of Chemistry and Chemical Engineering South China University of Technology(华南理工大学化学与化工学院);School of Biomedical Sciences The University of Hong Kong(香港大学生物医学科学学院);Computational Immunology Centre BayVax Biotech Limited(计算免疫学中心 BayVax 生物技术有限公司);College of Electronic and Information Engineering Southwest University(西南大学电子与信息工程学院)
Generalist-Specialist Mixture-of-Experts for Rare Pathology Detection in Multimodal Imaging
Technical University of Munich (TUM)(慕尼黑工业大学);TUM University Hospital(慕尼黑工业大学附属医院);Hasso Plattner Institute for Digital Engineering(哈索·普拉特纳数字工程研究所);University of Potsdam(波茨坦大学);Imperial College London(伦敦帝国理工学院)
MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference
School of Software, Tsinghua University, Beijing, China(清华大学软件学院,北京,中国);TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China(天津大学计算机科学与技术学院,中国);School of New Media and Communication, Tianjin University, China(天津大学新媒体与传播学院,中国)
Addressing Missing and Noisy Modalities in One Solution: Unified Modality-Quality Framework for Low-quality Multimodal Data
Department of Computer Science(计算机科学系);South China Normal University(华南师范大学);School of Electronics and Information Technology(电子与信息学院);Sun Yat-sen University(中山大学)
MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Zhongguancun Academy(中关村学院)
Hierarchical Time-Aware Mixture of Experts for Multi-Modal Sequential Recommendation
University of Science and Technology of China(中国科学技术大学);Nanjing University of Aeronautics and Astronautics(南京航空航天大学);School of Artificial Intelligence and Data Science, University of Science and Technology of China(中国科学技术大学人工智能与数据科学学院);Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州)人工智能学域);Department of Computer Science and Engineering, The Hong Kong University of Science and Technology(香港科技大学计算机科学与工程学系)
AlignMamba-2: Enhancing Multimodal Fusion and Sentiment Analysis with Modality-Aware Mamba
Pengcheng Laboratory(鹏城实验室);Shaanxi University of Science & Technology(陕西科技大学);School of Computer Science(计算机科学学院);Northwestern Polytechnical University(西北工业大学)
Flexible Multimodal Neuroimaging Fusion for Alzheimer's Disease Progression Prediction
Department of Computer Science and Engineering(计算机科学与工程系);Department of Biomedical Informatics(生物医学信息学系);Department of Neurology(神经病学系);Translational Data Analytics Institute(转化数据分析研究所)
BTW: A Non-Parametric Variance Stabilization Framework for Multimodal Model Integration
Department of Computer Science, Virginia Tech, Blacksburg, VA, USA(计算机科学系,弗吉尼亚理工学院,布莱克斯堡,VA,美国);Department of Agricultural and Applied Economics, Virginia Tech, Blacksburg, VA, USA(农业与应用经济学系,弗吉尼亚理工学院,布莱克斯堡,VA,美国)