arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2301.04408cs.CLcs.AIcs.CY
收藏

作为知识工作者的GPT:(AI)CPA能力的Zero-Shot评估

GPT as Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA Capabilities

Illinois Tech - Chicago Kent College of Law(伊利诺伊理工大学芝加哥肯特法学院);Bucerius Law School(布塞留斯法学院);CodeX - The Stanford Center for Legal Informatics(CodeX——斯坦福法律信息学中心)

Jillian Bommarito, Michael Bommarito, Daniel Martin Katz, Jessica Katz

摘要:

全球经济越来越依赖知识工作者来满足公共和私营组织的需求。虽然知识型工作没有统一的定义,但组织和行业团体仍然试图衡量个人从事知识型工作的能力。美国注册会计师协会(AICPA)制定的统一注册会计师考试是对专业知识型工作者能力准备程度最全面的评估。在本文中,我们对OpenAI的“text-davinci-003”和以前版本的GPT进行了实验性评估,包括监管(REG)考试样本和基于AICPA法律,财务,会计,技术和道德任务蓝图的200多个选择题的评估。首先,我们发现“text-davinci-003”在REG考试部分的正确率为14.4%,在zero-shot提示中的定量推理能力明显低于人类。第二,“text-davinci-003”在考试缺席计算中的记忆和理解以及应用技能水平上似乎接近人类水平。对于最佳提示和参数,该模型正确回答了57.6%的问题,显著优于25%的猜测率,并且其前两个答案正确率为82.1%,表明强非蕴涵。最后,我们发现,最近几代的GPT-3在这一评估上表现出实质性的改进,从“text-davinci-001”的30%上升到“text-davinci-003”的57%。这些发现有力地表明,大语言模型有潜力改变未来知识工作的质量和效率。

英文摘要:

The global economy is increasingly dependent on knowledge workers to meet the needs of public and private organizations. While there is no single definition of knowledge work, organizations and industry groups still attempt to measure individuals' capability to engage in it. The most comprehensive assessment of capability readiness for professional knowledge workers is the Uniform CPA Examination developed by the American Institute of Certified Public Accountants (AICPA). In this paper, we experimentally evaluate OpenAI's `text-davinci-003` and prior versions of GPT on both a sample Regulation (REG) exam and an assessment of over 200 multiple-choice questions based on the AICPA Blueprints for legal, financial, accounting, technology, and ethical tasks. First, we find that `text-davinci-003` achieves a correct rate of 14.4% on a sample REG exam section, significantly underperforming human capabilities on quantitative reasoning in zero-shot prompts. Second, `text-davinci-003` appears to be approaching human-level performance on the Remembering & Understanding and Application skill levels in the Exam absent calculation. For best prompt and parameters, the model answers 57.6% of questions correctly, significantly better than the 25% guessing rate, and its top two answers are correct 82.1% of the time, indicating strong non-entailment. Finally, we find that recent generations of GPT-3 demonstrate material improvements on this assessment, rising from 30% for `text-davinci-001` to 57% for `text-davinci-003`. These findings strongly suggest that large language models have the potential to transform the quality and efficiency of future knowledge work.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑