测量大型语言模型中的短形式事实性
Measuring short-form factuality in large language models
AI总结：
本文提出SimpleQA基准，用于评估语言模型回答简短事实问题的能力，强调挑战性和易评分性，并测试模型是否“知道自己知道什么”。
摘要：
我们提出了SimpleQA，这是一个基准测试，用于评估语言模型回答简短、寻求事实的问题的能力。在设计此评估时，我们优先考虑了两个属性。首先，SimpleQA具有挑战性，因为它是针对GPT-4的回复进行对抗性收集的。其次，回复易于评分，因为问题的设计确保只有一个无可争议的答案。SimpleQA中的每个答案都被评为正确、不正确或未尝试。理想行为的模型会尽可能多地正确回答问题，同时不尝试那些它不确定自己知道正确答案的问题。SimpleQA是一个简单、有针对性的评估，用于测试模型是否“知道自己知道什么”，我们希望这个基准测试在未来几代前沿模型中仍然具有相关性。SimpleQA可以在https://github.com/openai/simple-evals找到。
英文摘要：
We present SimpleQA, a benchmark that evaluates the ability of language models to answer short, fact-seeking questions. We prioritized two properties in designing this eval. First, SimpleQA is challenging, as it is adversarially collected against GPT-4 responses. Second, responses are easy to grade, because questions are created such that there exists only a single, indisputable answer. Each answer in SimpleQA is graded as either correct, incorrect, or not attempted. A model with ideal behavior would get as many questions correct as possible while not attempting the questions for which it is not confident it knows the correct answer. SimpleQA is a simple, targeted evaluation for whether models "know what they know," and our hope is that this benchmark will remain relevant for the next few generations of frontier models. SimpleQA can be found at https://github.com/openai/simple-evals.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
