大型语言模型的指令遵循评估
Instruction-Following Evaluation for Large Language Models
AI总结：
提出IFEval基准，通过25种可验证指令和约500个提示，客观评估LLM的指令遵循能力，并展示两个主流模型的评估结果。
摘要：
大型语言模型（LLMs）的核心能力之一是遵循自然语言指令。然而，对此类能力的评估尚未标准化：人工评估昂贵、缓慢且难以客观复现，而基于LLM的自动评估可能因评估器LLM的能力而存在偏见或受限。为克服这些问题，我们引入了用于大型语言模型的指令遵循评估（IFEval）。IFEval是一个直接且易于复现的评估基准，专注于一组“可验证指令”，例如“写超过400字”和“至少提及AI关键词3次”。我们识别了25种此类可验证指令，并构建了约500个提示，每个提示包含一个或多个可验证指令。我们展示了市场上两个广泛可用的LLM的评估结果。我们的代码和数据可在https://github.com/google-research/google-research/tree/master/instruction_following_eval获取。
英文摘要：
One core capability of Large Language Models (LLMs) is to follow natural language instructions. However, the evaluation of such abilities is not standardized: Human evaluations are expensive, slow, and not objectively reproducible, while LLM-based auto-evaluation is potentially biased or limited by the ability of the evaluator LLM. To overcome these issues, we introduce Instruction-Following Eval (IFEval) for large language models. IFEval is a straightforward and easy-to-reproduce evaluation benchmark. It focuses on a set of "verifiable instructions" such as "write in more than 400 words" and "mention the keyword of AI at least 3 times". We identified 25 types of those verifiable instructions and constructed around 500 prompts, with each prompt containing one or more verifiable instructions. We show evaluation results of two widely available LLMs on the market. Our code and data can be found at https://github.com/google-research/google-research/tree/master/instruction_following_eval
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
