尽管大语言模型(LLMs)在各类任务中取得了显著成就,但仍存在偏向英语等高资源语言的语言偏见,往往以牺牲低资源语言和区域语言为代价。为解决这一不平衡问题,我们提出了SeaLLMs,这是一个专门针对东南亚(SEA)语言的创新语言模型系列。SeaLLMs基于Llama-2模型构建,并通过扩展词表的持续预训练、专门的指令与对齐微调进一步提升,以更好地捕捉区域语言的细微之处。这使它们能够尊重并反映当地文化规范、习俗、风格偏好和法律考量。我们的综合评估表明,SeaLLM-13b模型在广泛的 linguistic 任务和助手式指令遵循能力上,相较于同类开源模型表现出更优越的性能。此外,它们在泰语、高棉语、老挝语和缅甸语等非拉丁语言上大幅超越ChatGPT-3.5,同时保持轻量化和成本效益高的运行特性。
英文摘要:
Despite the remarkable achievements of large language models (LLMs) in various tasks, there remains a linguistic bias that favors high-resource languages, such as English, often at the expense of low-resource and regional languages. To address this imbalance, we introduce SeaLLMs, an innovative series of language models that specifically focuses on Southeast Asian (SEA) languages. SeaLLMs are built upon the Llama-2 model and further advanced through continued pre-training with an extended vocabulary, specialized instruction and alignment tuning to better capture the intricacies of regional languages. This allows them to respect and reflect local cultural norms, customs, stylistic preferences, and legal considerations. Our comprehensive evaluation demonstrates that SeaLLM-13b models exhibit superior performance across a wide spectrum of linguistic tasks and assistant-style instruction-following capabilities relative to comparable open-source models. Moreover, they outperform ChatGPT-3.5 in non-Latin languages, such as Thai, Khmer, Lao, and Burmese, by large margins while remaining lightweight and cost-effective to operate.
A Novel Interpretability Metric for Explaining Bias in Language Models: Applications on Multilingual Models from Southeast Asia
School of Computer Science, University of Birmingham(英国伯明翰大学计算机科学学院);Department of Information Systems and Computer Science, Ateneo de Manila University(马尼拉大学信息系统与计算机科学系)
CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia
Singapore University of Technology and Design (SUTD)(新加坡科技设计大学);Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局);École Polytechnique Fédérale de Lausanne (EPFL)(洛桑联邦理工学院);Microsoft Research Asia (MSRA)(微软亚洲研究院);Sungkyunkwan University (SKKU)(成均馆大学);Universiti Brunei Darussalam (UBD)(文莱大学);China University of Petroleum (East China)(中国石油大学(华东));Nanyang Technological University (NTU)(南洋理工大学);Kyoto University(京都大学);Konkuk University(建国大学);Upstage AI
LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models
China-ASEAN Information Harbor Co., Ltd.(中国-东盟信息港有限公司);Beijing Academy of Artificial Intelligence(北京人工智能研究院);School of Software & Microelectronics, Peking University(北京大学软件与微电子学院)
SEA-CLIP-Tiny: Efficient Multilingual Text-Vision Embedding for Southeast Asian Languages
SEACrowd;University of Indonesia(印度尼西亚大学);Cohere Labs Community(Cohere Labs 社区);Technical University of Denmark(丹麦技术大学);Chulalongkorn University(朱拉隆功大学);Vidyasirimedhi Institute of Science and Technology(维迪亚西里梅迪科学技术研究所);AI Singapore(新加坡人工智能)
SEA-BED: How Do Embedding Models Represent Southeast Asian Languages?
Vidyasirimedhi Institute of Science and Technology(威差亚西里梅迪科学技术研究所);AI Singapore(新加坡人工智能);King Mongkut’s Institute of Technology Ladkrabang(拉卡邦先皇理工学院);Nanyang Technological University(南洋理工大学);Department of Computer Engineering, Faculty of Engineering, Chulalongkorn University(朱拉隆功大学工程学院计算机工程系)
Filipino Benchmarks for Measuring Sexist and Homophobic Bias in Multilingual Language Models from Southeast Asia
School of Computer Science, University of Birmingham(英国伯明翰大学计算机科学学院);Department of Information Systems and Computer Science, Ateneo de Manila University(马尼拉亚托大学信息系统与计算机科学系)
SAGE: Sustainable Agent-Guided Expert-tuning for Culturally Attuned Translation in Low-Resource Southeast Asia
Xi'an Jiaotong-Liverpool University(西安交通大学利物浦大学);University of Liverpool(利物浦大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
FilBench: Can LLMs Understand and Generate Filipino?
Allen Institute for AI(艾伦人工智能研究所);Nara Institute of Science and Technology(奈拉科学与技术研究所);Together AI;SEACrowd;MBZUAI(穆萨大学人工智能研究所);University of Bath(巴斯大学);National University, Philippines(菲律宾国家大学)
SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages
AI Singapore(新加坡人工智能中心);IndoNLP(印尼自然语言处理研究组);Universitas Indonesia(印度尼西亚大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);RevComm, Inc.(RevComm公司);Tohoku University(东北大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学);University of Bath(巴斯大学);National University Philippines(菲律宾国立大学);MOH Office for Healthcare Transformation (MOHT)(卫生部医疗转型办公室);NAIST(奈良科学技术研究所);Works Applications Lab(Works Applications实验室);Universitas Gadjah Mada(加查马达大学);Institut Teknologi Bandung(万隆理工学院);Beijing Academy of Artificial Intelligence (BAAI)(北京人工智能研究院);Institute for Infocomm Research, A*STAR(新加坡资讯通信研究院(新加坡科技研究局下属));Samsung Research Philippines(三星菲律宾研究院);University of Cambridge(剑桥大学);Queensland University of Technology(昆士兰科技大学)
Toward Robust Multilingual Adaptation of LLMs for Low-Resource Languages
Department of Automation, Tsinghua University, Beijing, China(清华大学自动化系);Alibaba International Digital Commerce Group, Beijing, China(阿里巴巴国际数字 commerce 集团);School of Software, Tsinghua University, Beijing, China(清华大学软件学院)
When Meaning Travels: A Granular Lens on Hybrid-MoE's Role in Idiomatic Understanding for Language Models
Department of Computer Science and Engineering, Indian Institute of Technology Patna, India(印度理工学院帕纳瓦分校计算机科学与工程系);School of Information Technology, King Mongkut’s Institute of Technology Ladkrabang, Thailand(泰国拉差班国王理工大学信息科技学院)
Anthropogenic Regional Adaptation in Multimodal Vision-Language Model
Cohere;SEACrowd;AI Singapore;Universiti Teknologi PETRONAS;Oracle;Carnegie Mellon University(卡内基梅隆大学);Monash University, Indonesia(莫纳什大学(印尼));King Mongkut’s University of Technology Thonburi(泰国孔敬大学);Nara Institute of Science and Technology(奈良科学技術大學);Stanford University(斯坦福大学);MBZUAI;National University of Singapore(新加坡国立大学);Monash University, Australia(莫纳什大学(澳大利亚));Brown University(布朗大学);University of Bath(巴斯大学);Mila - Quebec AI Institute(蒙特利尔AI研究所);Institut Teknologi Bandung(Bandung 工程技术大学);Ateneo de Manila University(马尼拉亚特内奥大学);Universitas Pelita Harapan(Pelita Harapan 大学);Seoul National University of Science and Technology(首尔科学技术大学);Thammasat University(泰国 Thammasat 大学);Independent(独立);University College London(伦敦大学学院);Nanyang Technological University(南洋理工大学);MiroMind AI;University of Indonesia(印度尼西亚大学);University of New Haven(纽黑文大学);INTI International University and Colleges(INTI 国际大学和学院);Binus University(Binus 大学);National University Philippines(菲律宾国家大学);ThoughtFull