arXiv 2503.017432025-03-07cs.CLcs.AIcs.LG

Phi-4-Mini 技术报告：通过 Mixture-of-LoRAs 实现紧凑而强大的多模态语言模型

Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs

Microsoft（微软）

Microsoft, :, Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congco… 展开作者

Microsoft, :, Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi-ling Chen, Qi Dai, Xiyang Dai, Ruchao Fan, Mei Gao, Min Gao, Amit Garg, Abhishek Goswami, Junheng Hao, Amr Hendy, Yuxuan Hu, Xin Jin, Mahmoud Khademi, Dongwoo Kim, Young Jin Kim, Gina Lee, Jinyu Li, Yunsheng Li, Chen Liang, Xihui Lin, Zeqi Lin, Mengchen Liu, Yang Liu, Gilsinia Lopez, Chong Luo, Piyush Madan, Vadim Mazalov, Arindam Mitra, Ali Mousavi, Anh Nguyen, Jing Pan, Daniel Perez-Becker, Jacob Platin, Thomas Portet, Kai Qiu, Bo Ren, Liliang Ren, Sambuddha Roy, Ning Shang, Yelong Shen, Saksham Singhal, Subhojit Som, Xia Song, Tetyana Sych, Praneetha Vaddamanu, Shuohang Wang, Yiming Wang, Zhenghao Wang, Haibin Wu, Haoran Xu, Weijian Xu, Yifan Yang, Ziyi Yang, Donghan Yu, Ishmam Zabir, Jianwen Zhang, Li Lyna Zhang, Yunan Zhang, Xiren Zhou

备注：: 39 pages

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出紧凑模型 Phi-4-Mini 和多模态模型 Phi-4-Multimodal，通过高质量合成数据与 Mixture-of-LoRAs 模态扩展，在数学、编码、语音等任务上达到或超越更大规模模型。

摘要：

我们介绍了 Phi-4-Mini 和 Phi-4-Multimodal，这是两款紧凑但能力很强的语言和多模态模型。Phi-4-Mini 是一个 38 亿参数的语言模型，基于高质量网络数据和合成数据训练，显著优于近期相似规模的开放源代码模型，并在需要复杂推理的数学和编码任务上达到其两倍规模模型的性能。这一成果得益于精心设计的合成数据配方，该配方强调高质量的数学和编码数据集。与其前身 Phi-3.5-Mini 相比，Phi-4-Mini 将词汇表扩展到 200K 个 token，以更好地支持多语言应用，并采用分组查询注意力机制以实现更高效的长序列生成。Phi-4-Multimodal 是一个多模态模型，将文本、视觉和语音/音频输入模态集成到单一模型中。其新颖的模态扩展方法利用 LoRA 适配器和模态特定路由器，使多种推理模式能够组合不同模态而不产生干扰。例如，尽管语音/音频模态的 LoRA 组件仅有 4.6 亿参数，该模型目前在 OpenASR 排行榜上排名第一。Phi-4-Multimodal 支持涉及（视觉 + 语言）、（视觉 + 语音）和（语音/音频）输入的场景，在广泛任务上优于更大的视觉-语言和语音-语言模型。此外，我们尝试进一步训练 Phi-4-Mini 以增强其推理能力。尽管其规模仅为紧凑的 38 亿参数，该实验版本在推理性能上与显著更大的模型持平或超越，包括 DeepSeek-R1-Distill-Qwen-7B 和 DeepSeek-R1-Distill-Llama-8B。

英文摘要：

We introduce Phi-4-Mini and Phi-4-Multimodal, compact yet highly capable language and multimodal models. Phi-4-Mini is a 3.8-billion-parameter language model trained on high-quality web and synthetic data, significantly outperforming recent open-source models of similar size and matching the performance of models twice its size on math and coding tasks requiring complex reasoning. This achievement is driven by a carefully curated synthetic data recipe emphasizing high-quality math and coding datasets. Compared to its predecessor, Phi-3.5-Mini, Phi-4-Mini features an expanded vocabulary size of 200K tokens to better support multilingual applications, as well as group query attention for more efficient long-sequence generation. Phi-4-Multimodal is a multimodal model that integrates text, vision, and speech/audio input modalities into a single model. Its novel modality extension approach leverages LoRA adapters and modality-specific routers to allow multiple inference modes combining various modalities without interference. For example, it now ranks first in the OpenASR leaderboard to date, although the LoRA component of the speech/audio modality has just 460 million parameters. Phi-4-Multimodal supports scenarios involving (vision + language), (vision + speech), and (speech/audio) inputs, outperforming larger vision-language and speech-language models on a wide range of tasks. Additionally, we experiment to further train Phi-4-Mini to enhance its reasoning capabilities. Despite its compact 3.8-billion-parameter size, this experimental version achieves reasoning performance on par with or surpassing significantly larger models, including DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。