GPQA：一个研究生级别的防Google问答基准
GPQA: A Graduate-Level Google-Proof Q&A Benchmark
AI总结：
本文提出了GPQA，一个由领域专家编写的448道多选题数据集，涵盖生物、物理和化学，旨在评估AI系统和非专家在研究生级别问题上的表现，为可扩展监督实验提供现实基础。
摘要：
我们提出了GPQA，一个包含448道多选题的极具挑战性的数据集，由生物学、物理学和化学领域的领域专家编写。我们确保问题具有高质量且极其困难：拥有或正在攻读相应领域博士学位的专家准确率达到65%（排除专家事后明确识别出的明显错误后为74%），而高技能非专家验证者尽管平均花费30多分钟且可不受限制地访问网络，准确率也仅为34%（即问题是“防Google”的）。这些问题对最先进的AI系统同样困难，我们基于GPT-4的最强基线准确率为39%。如果我们要利用未来的AI系统帮助回答非常困难的问题，例如在开发新科学知识时，我们需要开发可扩展的监督方法，使人类能够监督其输出，即使监督者本身具备技能和知识，这也可能很困难。GPQA对高技能非专家和前沿AI系统的困难性应能支持现实的可扩展监督实验，我们希望这能帮助设计出方法，使人类专家能可靠地从超越人类能力的AI系统中获取真实信息。
英文摘要：
We present GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. We ensure that the questions are high-quality and extremely difficult: experts who have or are pursuing PhDs in the corresponding domains reach 65% accuracy (74% when discounting clear mistakes the experts identified in retrospect), while highly skilled non-expert validators only reach 34% accuracy, despite spending on average over 30 minutes with unrestricted access to the web (i.e., the questions are "Google-proof"). The questions are also difficult for state-of-the-art AI systems, with our strongest GPT-4 based baseline achieving 39% accuracy. If we are to use future AI systems to help us answer very hard questions, for example, when developing new scientific knowledge, we need to develop scalable oversight methods that enable humans to supervise their outputs, which may be difficult even if the supervisors are themselves skilled and knowledgeable. The difficulty of GPQA both for skilled non-experts and frontier AI systems should enable realistic scalable oversight experiments, which we hope can help devise ways for human experts to reliably get truthful information from AI systems that surpass human capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
