测量大规模多任务语言理解
Measuring Massive Multitask Language Understanding
摘要:
我们提出了一个新的测试来衡量文本模型的多任务精度。该测试涵盖57项任务,包括小学数学,美国历史,计算机科学,法律等。为了在这个测试中获得高精度,模型必须拥有广泛的世界知识和解决问题的能力。我们发现,虽然大多数最近的模型具有接近随机机会的准确性,但最大的GPT-3模型平均比随机机会提高了近20个百分点。然而,在57项任务中的每一项任务上,最好的模型仍然需要进行实质性的改进,才能达到专家级的准确性。模型也有不平衡的表现,经常不知道什么时候他们错了。更糟糕的是,他们在道德和法律等社会重要问题上仍然具有近乎随机的准确性。通过全面评估模型的学术和专业理解的广度和深度,我们的测试可用于分析跨许多任务的模型,并识别重要的缺点。
英文摘要:
We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have near-random accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of a model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。