Recently developed large language models (LLMs) such as ChatGPT, Claude, and Llama have demonstrated impressive abilities, and even surpass human-level performance in several tasks. Despite their success, the resource-intensive demands of these models, requiring significant computational power for both training and inference, limit their deployment to high-performance servers. Additionally, the extensive calculation requirements of the models often lead to increased latency in response times. With the increasing need for LLMs to operate efficiently on CPUs, research about lightweight models that are optimized for CPU inference has emerged. In this work, we introduce GEB-1.3B, a lightweight LLM trained on 550 billion tokens in both Chinese and English languages. We employ novel training techniques, including ROPE, Group-Query-Attention, and FlashAttention-2, to accelerate training while maintaining model performance. Additionally, we fine-tune the model using 10 million samples of instruction data to enhance alignment. GEB-1.3B exhibits outstanding performance on general benchmarks such as MMLU, C-Eval, and CMMLU, outperforming comparative models such as MindLLM-1.3B and TinyLLaMA-1.1B. Notably, the FP32 version of GEB-1.3B achieves commendable inference times on CPUs, with ongoing efforts to further enhance speed through advanced quantization techniques. The release of GEB-1.3B as an open-source model marks a significant contribution to the development of lightweight LLMs, promising to foster further research and innovation in the field.
ALBETO and DistilBETO: Lightweight Spanish Language Models
University of Chile(智利大学);National Center for Artificial Intelligence (CENIA)(国家人工智能中心(CENIA));Millennium Institute for Foundational Research on Data (IMFD)(千禧数据基础研究所(IMFD));KU Leuven(鲁汶大学)
Analysing Lightweight Large Language Models for Biomedical Named Entity Recognition on Diverse Ouput Formats
Inria, Inserm, Université Paris Cité, HeKA, UMR 1346, Paris, France(法国国家科学研究中心(Inria)、法国国家医学研究院(Inserm)、巴黎城市大学(Université Paris Cité)、HeKA、法国国家科研机构(UMR 1346)、巴黎);Télécom Paris, Institut Polytechnique de Paris, France(巴黎电信学院(Télécom Paris)、巴黎理工学院(Institut Polytechnique de Paris)、法国)
A Lightweight Large Language Model-Based Multi-Agent System for 2D Frame Structural Analysis
Department of Civil and Architectural Engineering, University of Miami(迈阿密大学土木与建筑工程系);Department of Electrical and Computer Engineering, University of Miami(迈阿密大学电气与计算机工程系);College of Civil Engineering, Hunan University(湖南大学土木学院);Department of Computer Science, University of Illinois Chicago(伊利诺伊大学芝加哥分校计算机科学系);Department of Civil and Environmental Engineering, Washington State University(华盛顿州立大学土木与环境工程系);School of Architecture, University of Miami(迈阿密大学建筑学院)
Fine-tuning of lightweight large language models for sentiment classification on heterogeneous financial textual data
International Business School, Zhejiang University(浙江大学国际商务学院);Center for Data Science, Zhejiang University(浙江大学数据科学中心);Centre for Statistics, Georg-August-Universität Göttingen(哥廷根大学统计中心)
Exploring Lightweight Large Language Models for Court View Generation
School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);School of Computer Science, South China Normal University(华南师范大学计算机学院);China Mobile Internet Co., Ltd.(中国移动互联网有限公司)
Language Ranker: A Lightweight Ranking framework for LLM Decoding
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University(一般人工智能国家重点实验室,智能科学与技术学院,北京大学);Institute for Artificial Intelligence, Peking University(人工智能研究院,北京大学);MIT CSAIL, MA, USA(MIT CSAIL,马萨诸塞州,美国);Microsoft Research Asia(微软亚洲研究院)
Lightweight Retrieval-Augmented Generation and Large Language Model-Based Modeling for Scalable Patient-Trial Matching
Department of Artificial Intelligence and Informatics, Mayo Clinic(人工智能与信息学系,梅奥诊所);Computer Science Department, University of Tulsa(图兰大学计算机科学系);Mayo Clinic Comprehensive Cancer Center, Mayo Clinic(梅奥诊所综合癌症中心,梅奥诊所);Division of Community Pediatric and Adolescent Medicine, Department of Pediatrics, Mayo Clinic(社区儿科与青少年医学分会,儿科部,梅奥诊所);Department of Neurology, Mayo Clinic(神经病学部,梅奥诊所);Department of Radiation Oncology, Mayo Clinic(放射肿瘤学部,梅奥诊所);Department of Quantitative Health Sciences, Mayo Clinic(定量健康科学部,梅奥诊所)
PokerGPT: An End-to-End Lightweight Solver for Multi-Player Texas Hold'em via Large Language Model
University of Electronic Science and Technology of China(电子科技大学);Shenzhen Institute of Advanced Study, UESTC, China(电子科技大学深圳高等研究院);Sichuan Optical Character Technology Co., Ltd(四川光学字符技术有限公司)
Performance Evaluation of Lightweight Open-source Large Language Models in Pediatric Consultations: A Comparative Analysis
Children’s Hospital of Chongqing Medical University(重庆医科大学附属儿童医院);Stanford University School of Medicine(斯坦福大学医学院);Chongqing University of Technology(重庆理工大学);Columbia University(哥伦比亚大学)