arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.01723cs.CLcs.AI
收藏

中文工业场景下大语言模型准确性与鲁棒性的实证研究

An Empirical Study on Large Language Models in Accuracy and Robustness under Chinese Industrial Scenarios

Hong Kong University of Science and Technology(香港科技大学);China Academy of Industrial Internet(中国工业互联网研究院)

Zongjie Li, Wenying Qiu, Pingchuan Ma, Yichen Li, You Li, Sijia He, Baozheng Jiang, Shuai Wang, Weixi Gu

AI总结:

本研究通过收集1,200个工业问题和13,631个鲁棒性测试变体,实证评估了13种大语言模型,发现其在中文工业场景下准确性低且鲁棒性因领域和能力而异,本地模型总体弱于全球模型。

摘要:

近年来,大语言模型(LLM)在各个领域得到了快速发展。为了更好地服务众多中文用户,中国许多商业供应商采取了本地化策略,训练并提供专门针对中文用户定制的本地大语言模型。此外,展望未来,大语言模型的关键应用之一将是企业和用户在工业生产中的实际部署。然而,大语言模型在工业场景中的准确性和鲁棒性尚未得到充分研究。本文针对中文工业生产领域,对大语言模型的准确性和鲁棒性进行了全面的实证研究。我们手动收集了来自8个不同工业领域的1,200个领域特定问题,以评估大语言模型的准确性。此外,我们设计了一个蜕变测试框架,包含四个工业特定稳定性类别和八种能力,共计13,631个带变体的问题,以评估大语言模型的鲁棒性。我们总共评估了由中国供应商开发的9种不同大语言模型,以及由全球供应商开发的4种不同大语言模型。我们的主要发现包括:(1)当前大语言模型在中文工业情境下准确性较低,所有大语言模型的得分均低于0.6。(2)鲁棒性得分在不同工业领域间存在差异,本地大语言模型总体表现差于全球大语言模型。(3)大语言模型的鲁棒性在不同能力间差异显著。全球大语言模型在逻辑相关变体下更为鲁棒,而先进的本地大语言模型在理解中文工业术语相关问题上表现更好。我们的研究结果为从开发和工业企业角度理解和提升大语言模型的工业领域能力提供了有价值的指导。研究结果进一步激发了可能的研究方向和工具支持。

英文摘要:

Recent years have witnessed the rapid development of large language models (LLMs) in various domains. To better serve the large number of Chinese users, many commercial vendors in China have adopted localization strategies, training and providing local LLMs specifically customized for Chinese users. Furthermore, looking ahead, one of the key future applications of LLMs will be practical deployment in industrial production by enterprises and users in those sectors. However, the accuracy and robustness of LLMs in industrial scenarios have not been well studied. In this paper, we present a comprehensive empirical study on the accuracy and robustness of LLMs in the context of the Chinese industrial production area. We manually collected 1,200 domain-specific problems from 8 different industrial sectors to evaluate LLM accuracy. Furthermore, we designed a metamorphic testing framework containing four industrial-specific stability categories with eight abilities, totaling 13,631 questions with variants to evaluate LLM robustness. In total, we evaluated 9 different LLMs developed by Chinese vendors, as well as four different LLMs developed by global vendors. Our major findings include: (1) Current LLMs exhibit low accuracy in Chinese industrial contexts, with all LLMs scoring less than 0.6. (2) The robustness scores vary across industrial sectors, and local LLMs overall perform worse than global ones. (3) LLM robustness differs significantly across abilities. Global LLMs are more robust under logical-related variants, while advanced local LLMs perform better on problems related to understanding Chinese industrial terminology. Our study results provide valuable guidance for understanding and promoting the industrial domain capabilities of LLMs from both development and industrial enterprise perspectives. The results further motivate possible research directions and tooling support.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑