人物角色增强基准测试：评估大语言模型在不同写作风格下的表现
Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Styles
AI总结：
本研究通过人物角色提示重写评估提示，发现写作风格显著影响LLM性能，并提出可扩展的基准增强方法以提升评估外部有效性。
摘要：
当前用于评估大语言模型（LLMs）的基准测试往往缺乏足够的写作风格多样性，许多基准主要遵循标准化惯例。这类基准未能充分捕捉人类所展现的丰富多样的交流模式。因此，在这些基准上优化的大语言模型在面对“非标准”输入时，可能表现出脆弱的性能。在本工作中，我们通过使用基于人物角色的LLM提示重写评估提示来检验这一假设，这是一种模拟多样写作风格的低成本方法。我们的结果表明，即使语义内容相同，写作风格和提示格式的变化也会显著影响被评估LLM的估计性能。值得注意的是，我们识别出不同的写作风格，这些风格在多种模型和任务上持续触发低或高的性能，与模型家族、规模和发布时间无关。我们的工作提供了一种可扩展的方法来增强现有基准，提高了这些基准在衡量LLM跨语言变体性能方面所提供评估的外部有效性。
英文摘要：
Current benchmarks for evaluating Large Language Models (LLMs) often do not exhibit enough writing style diversity, with many adhering primarily to standardized conventions. Such benchmarks do not fully capture the rich variety of communication patterns exhibited by humans. Thus, it is possible that LLMs, which are optimized on these benchmarks, may demonstrate brittle performance when faced with "non-standard" input. In this work, we test this hypothesis by rewriting evaluation prompts using persona-based LLM prompting, a low-cost method to emulate diverse writing styles. Our results show that, even with identical semantic content, variations in writing style and prompt formatting significantly impact the estimated performance of the LLM under evaluation. Notably, we identify distinct writing styles that consistently trigger either low or high performance across a range of models and tasks, irrespective of model family, size, and recency. Our work offers a scalable approach to augment existing benchmarks, improving the external validity of the assessments they provide for measuring LLM performance across linguistic variations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
