The rapid evolution of large language models (LLMs) has transformed the competitive landscape in natural language processing (NLP), particularly for English and other data-rich languages. However, underrepresented languages like Cantonese, spoken by over 85 million people, face significant development gaps, which is particularly concerning given the economic significance of the Guangdong-Hong Kong-Macau Greater Bay Area, and in substantial Cantonese-speaking populations in places like Singapore and North America. Despite its wide use, Cantonese has scant representation in NLP research, especially compared to other languages from similarly developed regions. To bridge these gaps, we outline current Cantonese NLP methods and introduce new benchmarks designed to evaluate LLM performance in factual generation, mathematical logic, complex reasoning, and general knowledge in Cantonese, which aim to advance open-source Cantonese LLM technology. We also propose future research directions and recommended models to enhance Cantonese LLM development.
DependEval: Benchmarking LLMs for Repository Dependency Understanding
Nanyang Technological University(南洋理工大学);Tsinghua University(清华大学);Beihang University(北京航空航天大学);ShanghaiTech University(上海科技大学);The University of Hong Kong(香港大学)
BAGEL: Benchmarking Animal Knowledge Expertise in Language Models
Earth Species Project(地球物种项目);NYU Center for Data Science(纽约大学数据科学中心;纽约大学上海);NYU Shanghai(纽约大学上海数据科学中心;纽约大学-复旦大学数学科学研究所);NYU Shanghai Center for Data Science;NYU-ECNU Institute of Mathematical Sciences at NYU Shanghai
IPBench: Benchmarking the Knowledge of Large Language Models in Intellectual Property
Shenzhen Key Laboratory for High Performance Data Mining, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, China(深圳高性能数据挖掘重点实验室,深圳先进技术研究院,中国科学院,中国);University of Chinese Academy of Sciences, China(中国科学院大学,中国);Dalian University of Technology, China(大连理工大学,中国);School of Biomedical Engineering, UNSW Sydney, Australia(生物医学工程学院,新南威尔士大学悉尼分校,澳大利亚);Shenzhen MSU-BIT University, China(深圳MSU-BIT大学,中国);Shenzhen University of Advanced Technology, China(深圳大学先进技术学院,中国)
Vote'n'Rank: Revision of Benchmarking with Social Choice Theory
HSE University(高等经济大学);SaluteDevices(SaluteDevices公司);University of Oxford(牛津大学);Artificial Intelligence Research Institute(人工智能研究院);Institute of Linguistics, RAS(俄罗斯科学院语言学研究所);Center for Information and Language Processing, LMU Munich(慕尼黑大学信息与语言处理中心)
CUB: Benchmarking Context Utilisation Techniques for Language Models
Chalmers University of Technology(查尔姆斯理工大学);University of Gothenburg(哥德堡大学);Seoul National University(首尔国立大学);University of Copenhagen(哥本哈根大学);Ewha Womans University(成均馆大学)