We introduce the Yi model family, a series of language and multimodal models that demonstrate strong multi-dimensional capabilities. The Yi model family is based on 6B and 34B pretrained language models, then we extend them to chat models, 200K long context models, depth-upscaled models, and vision-language models. Our base models achieve strong performance on a wide range of benchmarks like MMLU, and our finetuned chat models deliver strong human preference rate on major evaluation platforms like AlpacaEval and Chatbot Arena. Building upon our scalable super-computing infrastructure and the classical transformer architecture, we attribute the performance of Yi models primarily to its data quality resulting from our data-engineering efforts. For pretraining, we construct 3.1 trillion tokens of English and Chinese corpora using a cascaded data deduplication and quality filtering pipeline. For finetuning, we polish a small scale (less than 10K) instruction dataset over multiple iterations such that every single instance has been verified directly by our machine learning engineers. For vision-language, we combine the chat language model with a vision transformer encoder and train the model to align visual representations to the semantic space of the language model. We further extend the context length to 200K through lightweight continual pretraining and demonstrate strong needle-in-a-haystack retrieval performance. We show that extending the depth of the pretrained checkpoint through continual pretraining further improves performance. We believe that given our current results, continuing to scale up model parameters using thoroughly optimized data will lead to even stronger frontier models.
EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training
Institute of Computer Science, University of Tartu（塔尔图大学计算机科学研究院）；Department of Software Science, Tallinn University of Technology（塔林技术大学软件科学系）；Institute of the Estonian Language, Tallinn, Estonia（爱沙尼亚语言研究院）；School of Humanities, Tallinn University（塔林大学人文学院）
ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data
Beijing University of Posts and Telecommunications（北京邮电大学）；Pengcheng Laboratory（鹏城实验室）；China Unicom Network Communications Corporation Limited（中国联合网络通信股份有限公司）
A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT
Michigan State University（密歇根州立大学）；Lehigh University（理海大学）；Beihang University（北京航空航天大学）；Macquarie University（麦考瑞大学）；Nanyang Technological University（南洋理工大学）；University of California San Diego（加利福尼亚大学圣地亚哥分校）；Salesforce AI Research（Salesforce人工智能研究院）；Duke University（杜克大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
STEP: Scientific Time-Series Encoder Pretraining via Cross-Domain Distillation
Shanghai Aritificial Intelligence Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）；Harbin Institute of Technology（哈尔滨工业大学）；Sichuan University（四川大学）；University of Cambridge（剑桥大学）
On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective
Microsoft Research（微软研究院）；City University of Hong Kong（香港城市大学）；Microsoft STCA（微软中国区软件工程技术中心）；Carnegie Mellon University（卡内基梅隆大学）；Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Peking University（北京大学）；Westlake University（西湖大学）
OntoChatGPT Information System: Ontology-Driven Structured Prompts for ChatGPT Meta-Learning
Glushkov Institute of Cybernetics of the National Academy of Sciences of Ukraine（乌克兰国家科学院格鲁什科夫控制论研究所）；Frantsevic Institute for Problems in Material Science of the National Academy of Sciences of Ukraine（乌克兰国家科学院弗兰采维奇材料科学问题研究所）
Few-Shot Learning of an Interleaved Text Summarization Model by Pretraining with Synthetic Data
Center for Information and Language Processing (CIS), LMU Munich（慕尼黑大学信息与语言处理中心（CIS））；Machine Intelligence, Siemens Healthineers, Princeton（西门子医疗 Princeton 机器智能部门）；Toyota Research Institute, Los Altos, California（丰田研究所）；Machine Intelligence, Siemens AG, Munich（西门子股份公司慕尼黑机器智能部门）
Don't Stop Pretraining: Adapt Language Models to Domains and Tasks
Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）
