Large language models (LLMs) have demonstrated prowess in a wide range of tasks. However, many LLMs exhibit significant performance discrepancies between high- and low-resource languages. To mitigate this challenge, we present FuxiTranyu, an open-source multilingual LLM, which is designed to satisfy the need of the research community for balanced and high-performing multilingual capabilities. The base model, FuxiTranyu-8B, features 8 billion parameters and is trained from scratch on meticulously balanced multilingual data that contains 600 billion tokens covering 43 natural languages and 16 programming languages. We also develop two instruction-tuned models: FuxiTranyu-8B-SFT which is fine-tuned on a diverse multilingual instruction dataset, and FuxiTranyu-8B-DPO which is further refined with DPO on a preference dataset for enhanced alignment ability. Extensive experiments on a wide range of multilingual benchmarks demonstrate the competitive performance of FuxiTranyu against existing multilingual LLMs, e.g., BLOOM-7B, PolyLM-13B, and Mistral-7B-Instruct. Both neuron and representation interpretability analyses reveal that FuxiTranyu achieves consistent multilingual representations across languages. To promote further research into multilingual LLMs, we release both the base and instruction-tuned FuxiTranyu models together with 58 pre-training checkpoints at HuggingFace (see https://huggingface.co/TJUNLP/FuxiTranyu-8B) and Github (see https://github.com/tjunlp-lab/FuxiTranyu).
AlignX: Advancing Multilingual Large Language Models with Multilingual Representation Alignment
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)(智能信息处理重点实验室,计算技术研究所,中国科学院);Key Laboratory of AI Safety, Chinese Academy of Sciences(人工智能安全重点实验室,中国科学院);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京,中国);Baidu Inc.(百度公司)
Language on Demand, Knowledge at Core: Composing LLMs with Encoder-Decoder Translation Models for Extensible Multilinguality
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences(智能信息处理重点实验室,计算技术研究所,中国科学院);State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(人工智能安全国家重点实验室,计算技术研究所,中国科学院);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京,中国)
Reliable Multilingual Orthopedic Decision Support from Clinical Narratives: Language-Aware Adaptation and Verification-Guided Deferral
School of Computer Science, Wuhan University(武汉大学计算机学院);School of Software, Nanjing University of Information Science and Technology (NUIST)(南京信息工程大学软件学院);Department of Orthopedic, Bahawal Victoria Hospital(巴哈瓦尔维多利亚医院骨科部门)
SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning
Nanyang Technological University(南洋理工大学);Institute for Infocomm Research (IR), A*STAR(新加坡科技研究局信息通信研究所);Centre for Frontier AI Research (CFAR), A*STAR(新加坡科技研究局前沿人工智能研究中心)
EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training
Institute of Computer Science, University of Tartu(塔尔图大学计算机科学研究院);Department of Software Science, Tallinn University of Technology(塔林技术大学软件科学系);Institute of the Estonian Language, Tallinn, Estonia(爱沙尼亚语言研究院);School of Humanities, Tallinn University(塔林大学人文学院)
Language Exposure Matters: Distribution and Order in Multilingual BabyLM Pre-training
School of Computation, Information and Technology, TU Munich(慕尼黑工业大学计算、信息与技术学院);Munich Center for Machine Learning(慕尼黑机器学习中心);Munich Data Science Institute(慕尼黑数据科学研究所)
MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining
Hong Kong University of Science and Technology (Guangzhou)(香港理工大学(广州));ByteDance(字节跳动);Eindhoven University of Technology(埃因霍温理工大学);University of Melbourne(墨尔本大学);University of Liverpool(利物浦大学)
BenSyc: Benchmarking Conversational Sycophancy and Human Alignment in LLMs for Bengali Contexts
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Virginia Tech(弗吉尼亚理工大学);Bangladesh University of Engineering and Technology(孟加拉工程与技术大学);BRAC University(BRAC大学);University of Toronto(多伦多大学)
Semi-automatic Generation of Multilingual Datasets for Stance Detection in Twitter
Vicomtech Foundation(维康泰克基金会);Basque Research and Technology Alliance (BRTA)(巴斯克研究与技术联盟(BRTA));University of the Basque Country UPV/EHU(巴斯克大学(UPV/EHU))