考虑不同指令模板间分数方差的大型语言模型评估方法研究
Toward the Evaluation of Large Language Models Considering Score Variance across Instruction Templates
AI总结：
针对现有评估方法忽略提示词差异导致不公平的问题，本研究提供英日跨语言数据集并提出考虑模板间分数方差的Sharpe score指标，揭示了模板高方差对LLMs公平评估的显著影响。
摘要：
大型语言模型（LLMs）的自然语言理解（NLU）性能已在各种任务和数据集上得到了评估。然而，现有的评估方法没有考虑到由于提示词差异导致的分数方差，这导致了对NLU性能的不公平评估和比较。此外，针对特定提示词设计的评估不适合指令微调，后者的目标是在任何提示词下都能表现良好。因此，有必要找到一种公平的方法来衡量NLU性能，并考虑不同指令模板之间的分数方差。在本研究中，我们提供了英语和日语的跨语言数据集，用于评估LLMs的NLU性能，其中包含用于公平评估每个任务的多个指令模板，以及用于约束输出格式的正则表达式。此外，我们提出了Sharpe score作为一种评估指标，它考虑了模板之间分数的方差。对英语和日语LLMs的综合分析表明，模板之间的高方差对LLMs的公平评估具有显著影响。
英文摘要：
The natural language understanding (NLU) performance of large language models (LLMs) has been evaluated across various tasks and datasets. The existing evaluation methods, however, do not take into account the variance in scores due to differences in prompts, which leads to unfair evaluation and comparison of NLU performance. Moreover, evaluation designed for specific prompts is inappropriate for instruction tuning, which aims to perform well with any prompt. It is therefore necessary to find a way to measure NLU performance in a fair manner, considering score variance between different instruction templates. In this study, we provide English and Japanese cross-lingual datasets for evaluating the NLU performance of LLMs, which include multiple instruction templates for fair evaluation of each task, along with regular expressions to constrain the output format. Furthermore, we propose the Sharpe score as an evaluation metric that takes into account the variance in scores between templates. Comprehensive analysis of English and Japanese LLMs reveals that the high variance among templates has a significant impact on the fair evaluation of LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
