arXiv 2404.142192024-08-30cs.CLcs.AI

Phi-3 技术报告：一款可在手机上本地运行的高性能语言模型

Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

Microsoft（微软）

Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim,… 展开作者

Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, Xiren Zhou

备注：: 24 pages

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文报告了 Phi-3 系列小型语言模型，通过高质量数据训练和参数扩展，使手机端可部署的模型在多项基准上达到或超越更大规模模型，并进一步推出 phi-3.5 系列以增强多语言、多模态和长上下文能力。

摘要：

我们介绍 phi-3-mini，这是一个拥有 38 亿参数、在 3.3 万亿 token 上训练的语言模型。无论是通过学术基准测试还是内部测试来衡量，其整体性能均可与 Mixtral 8x7B 和 GPT-3.5 等模型相媲美（例如，phi-3-mini 在 MMLU 上达到 69%，在 MT-bench 上达到 8.38），尽管其体积小到足以部署在手机上。我们的训练数据集是 phi-2 所用数据集的扩展版本，由经过严格过滤的公开网络数据和合成数据组成。该模型还进一步针对鲁棒性、安全性和聊天格式进行了对齐。我们还提供了参数扩展结果，包括在 4.8T token 上训练的 7B 和 14B 模型，分别称为 phi-3-small 和 phi-3-medium，两者均显著强于 phi-3-mini（例如，在 MMLU 上分别达到 75% 和 78%，在 MT-bench 上分别达到 8.7 和 8.9）。为了增强多语言、多模态和长上下文能力，我们引入了 phi-3.5 系列中的三个模型：phi-3.5-mini、phi-3.5-MoE 和 phi-3.5-Vision。phi-3.5-MoE 是一个 16 x 3.8B 的 MoE 模型，拥有 66 亿激活参数，在语言推理、数学和代码任务上，相比其他同等规模的开源模型（如 Llama 3.1 和 Mixtral 系列）取得了更优的性能，并与 Gemini-1.5-Flash 和 GPT-4o-mini 相当。同时，phi-3.5-Vision 是一个源自 phi-3.5-mini 的 42 亿参数模型，在推理任务上表现出色，并且擅长处理单图像与文本提示，以及多图像与文本提示。

英文摘要：

We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. Our training dataset is a scaled-up version of the one used for phi-2, composed of heavily filtered publicly available web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide parameter-scaling results with a 7B, 14B models trained for 4.8T tokens, called phi-3-small, phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75%, 78% on MMLU, and 8.7, 8.9 on MT-bench). To enhance multilingual, multimodal, and long-context capabilities, we introduce three models in the phi-3.5 series: phi-3.5-mini, phi-3.5-MoE, and phi-3.5-Vision. The phi-3.5-MoE, a 16 x 3.8B MoE model with 6.6 billion active parameters, achieves superior performance in language reasoning, math, and code tasks compared to other open-source models of similar scale, such as Llama 3.1 and the Mixtral series, and on par with Gemini-1.5-Flash and GPT-4o-mini. Meanwhile, phi-3.5-Vision, a 4.2 billion parameter model derived from phi-3.5-mini, excels in reasoning tasks and is adept at handling both single-image and text prompts, as well as multi-image and text prompts.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。