Recent years have witnessed the rapid development of large language models (LLMs) in various domains. To better serve the large number of Chinese users, many commercial vendors in China have adopted localization strategies, training and providing local LLMs specifically customized for Chinese users. Furthermore, looking ahead, one of the key future applications of LLMs will be practical deployment in industrial production by enterprises and users in those sectors. However, the accuracy and robustness of LLMs in industrial scenarios have not been well studied. In this paper, we present a comprehensive empirical study on the accuracy and robustness of LLMs in the context of the Chinese industrial production area. We manually collected 1,200 domain-specific problems from 8 different industrial sectors to evaluate LLM accuracy. Furthermore, we designed a metamorphic testing framework containing four industrial-specific stability categories with eight abilities, totaling 13,631 questions with variants to evaluate LLM robustness. In total, we evaluated 9 different LLMs developed by Chinese vendors, as well as four different LLMs developed by global vendors. Our major findings include: (1) Current LLMs exhibit low accuracy in Chinese industrial contexts, with all LLMs scoring less than 0.6. (2) The robustness scores vary across industrial sectors, and local LLMs overall perform worse than global ones. (3) LLM robustness differs significantly across abilities. Global LLMs are more robust under logical-related variants, while advanced local LLMs perform better on problems related to understanding Chinese industrial terminology. Our study results provide valuable guidance for understanding and promoting the industrial domain capabilities of LLMs from both development and industrial enterprise perspectives. The results further motivate possible research directions and tooling support.
A Large-Scale Chinese Knowledge Graph-Text Alignment Dataset for Benchmarking Knowledge-Grounded LLMs
Nanjing University of Science and Techonolgy(南京理工大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);University of Science and Technology Beijing(北京科技大学);Hefei University of Technology(合肥工业大学);Beijing University of Chemical Technology(北京化工大学);Renmin University of China(中国人民大学);Beihang University(北航);Beijing University of Posts and Telecommunications(北京邮电大学);Griffith University, Australia(澳大利亚格里菲斯大学)
Enhance Robustness of Language Models Against Variation Attack through Graph Integration
Institute for Intelligent Computing, Alibaba Group(阿里巴巴集团智能计算研究院);School of Information Management, Wuhan University(武汉大学信息管理学院);Worcester Polytechnic Institute(伍斯特理工学院)
Evaluating Chinese Ambiguity Understanding in Large Language Models
Graduate School of Information Science and Technology, The University of Tokyo(东京大学信息科学与技术研究生院);School of Software Engineering, South China University of Technology(华南理工大学软件学院)
CNSL-bench: Benchmarking the Sign Language Understanding Capabilities of MLLMs on Chinese National Sign Language
School of Informatics, Xiamen University, China(厦门大学信息学院);Key Lab of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian-Taiwan (XMU), Ministry of Culture and Tourism, China(福建省-台湾非物质文化遗产数字化保护与智能处理重点实验室(XMU),文化和旅游部,中国);National Language Resources Monitoring and Research Center for Education and Teaching Media, Xiamen University, China(教育与教学媒体语言资源监测与研究中心,厦门大学,中国)
Language-Agnostic Suicidal Risk Detection Using Large Language Models
Wonkwang University HospitalRepublic of Korea(Wonkwang大学医院);MODULABSRepublic of Korea(MODULABS);Wonkwang UniversityRepublic of Korea(Wonkwang大学);Department of Psychiatry RSC LAB(精神病学系RSC实验室);Department of Psychiatry, School of Medicine(精神病学系,医学院)
Large Language Model for Extracting Complex Contract Information in Industrial Scenes
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University, Shanghai, China(上海智能自主系统研究院,同济大学);School of Computer Science and Technology, East China Normal University, Shanghai, China(计算机科学与技术学院,华东师范大学)
LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound
ZJU-UIUC Institute, Zhejiang University(浙江大学ZJU-UIUC学院);University of Washington(华盛顿大学);Zhejiang University School of Medicine Sir Run Run Shaw Hospital(浙江大学医学院附属邵逸夫医院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
RobustMAD: Evaluating Real-World Robustness of Multimodal Small Language Models for Deployable Anomaly Detection Assistants
Singapore University of Technology and Design(新加坡科技设计大学);Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局);Nanyang Technological University(南洋理工大学);Chongqing University(重庆大学)
Efficient Training of Robust Traditional Chinese LLaMA-1B on a Single Consumer GPU: Continual Pre-training, SFT, and DPO
National Tsing Hua University(台湾清华大学);Accton Technology Corporation(智邦科技公司);National Yang Ming Chiao Tung University(阳明交通大学);University of Taipei(台北市立大学)