arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.06341cs.LGcs.AIcs.SE
收藏

评估大型语言模型在企业应用中的鲁棒性:跨格式和语言的扰动一致性基准测试

Evaluating Robustness of Large Language Models in Enterprise Applications: Benchmarks for Perturbation Consistency Across Formats and Languages

ServiceNow

Tara Bogavelli, Oluwanifemi Bamgbose, Gabrielle Gauthier Melançon, Fanny Riols, Roshnee Sharma

AI总结:

本研究提出跨格式和语言的扰动一致性基准测试,评估大型语言模型在企业应用中的鲁棒性,发现模型大小与鲁棒性关系复杂,8B参数模型表现各异。

摘要:

企业大型语言模型应用需要在多样化场景中保持一致的高质量和可靠性能,要求对微小变化具有鲁棒性。现有研究显示,即使是小的提示变化也可能导致输出的显著差异,但主要集中在狭窄的扰动集和小学术数据集上,限制了其在现实应用中的相关性。为此,我们提出了一套综合的基准测试套件,评估多种扰动类型,包括通用文本编辑(如标点、空格)、格式变化(如JSON、YAML)、多语言和跨语言输入以及指令中的位置变化。评估了从4B到120B+参数的11个模型,发现微小扰动会使关键企业指标的性能下降高达40个百分点。关键的是,我们展示了模型大小与鲁棒性之间的关系比传统假设更复杂:一个8B参数模型(Ministral 3 8B)在大多数较大模型之上表现更好,而另一个8B模型(Llama 3.1 8B)总体表现最差。

英文摘要:

Enterprise LLM applications require consistently high quality and reliable performance across diverse scenarios, demanding robustness to minor variations. Existing research shows that even small prompt changes can lead to substantial differences in output, but has mainly focused on a narrow set of perturbations with small academic datasets, limiting their relevance to real-world applications. To address this, we present a comprehensive benchmark suite that evaluates robustness across multiple perturbation types, including general text edits (e.g., punctuation, whitespace), formatting changes (e.g., JSON, YAML), multilingual and cross-lingual inputs, and positional variations in instructions. Evaluating 11 models ranging from 4B to 120B+ parameters, we find that minor perturbations reduce performance by up to 40 percentage points on key enterprise metrics. Critically, we demonstrate that the relationship between model size and robustness is more nuanced than conventional assumptions suggest: an 8B parameter model (Ministral 3 8B) outperforms most larger models, while another 8B model (Llama 3.1 8B) performs worst overall.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑