生成式人工智能时代大型语言模型基准测试的不足
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
AI总结：
本文用统一框架从人员、流程、技术维度评审23个LLM基准，揭示偏见、推理测量和一致性等缺陷，呼吁转向动态行为画像与标准化评估。
摘要：
具备新兴能力的大型语言模型（LLM）迅速普及，激发了公众评估和比较不同 LLM 的好奇心，促使许多研究人员提出各自的 LLM 基准。注意到这些基准存在初步不足，我们开展了一项研究，批判性评估 23 个最先进的 LLM 基准；研究采用我们新颖的统一评估框架，在基准功能性与诚信两大支柱下，从人员、流程和技术三个视角进行考察。研究揭示了显著局限，包括偏见、难以衡量真正推理、适应性、实施不一致、提示工程复杂性、评估者多样性，以及在一次综合评估中忽视文化和意识形态规范。我们的讨论强调，面对人工智能（AI）进步，迫切需要标准化方法、监管确定性和伦理准则，并倡导从静态基准演进到动态行为画像，以准确捕捉 LLM 的复杂行为和潜在风险。本研究凸显了 LLM 评估方法范式转变的必要性，强调应协同努力开发普遍接受的基准，并增强 AI 系统与社会的融合。
英文摘要：
The rapid rise in popularity of Large Language Models (LLMs) with emerging capabilities has spurred public curiosity to evaluate and compare different LLMs, leading many researchers to propose their own LLM benchmarks. Noticing preliminary inadequacies in those benchmarks, we embarked on a study to critically assess 23 state-of-the-art LLM benchmarks, using our novel unified evaluation framework through the lenses of people, process, and technology, under the pillars of benchmark functionality and integrity. Our research uncovered significant limitations, including biases, difficulties in measuring genuine reasoning, adaptability, implementation inconsistencies, prompt engineering complexity, evaluator diversity, and the overlooking of cultural and ideological norms in one comprehensive assessment. Our discussions emphasized the urgent need for standardized methodologies, regulatory certainties, and ethical guidelines in light of Artificial Intelligence (AI) advancements, including advocating for an evolution from static benchmarks to dynamic behavioral profiling to accurately capture LLMs' complex behaviors and potential risks. Our study highlighted the necessity for a paradigm shift in LLM evaluation methodologies, underlining the importance of collaborative efforts for the development of universally accepted benchmarks and the enhancement of AI systems' integration into society.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
