arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.17003cs.CLcs.AIcs.LG
收藏

在巴西大学入学考试中评估GPT-3 .5和GPT-4模型

Evaluating GPT-3.5 and GPT-4 Models on Brazilian University Admission Exams

University of São Paulo (USP)(圣保罗大学);University of San Francisco (USF)(旧金山大学);Maritaca AI(Maritaca人工智能公司);NeuralMind(NeuralMind公司);State University of Campinas (UNICAMP)(坎皮纳斯州立大学)

Desnes Nunes, Ricardo Primi, Ramon Pires, Roberto Lotufo, Rodrigo Nogueira

摘要:

本研究旨在探讨语言模型(LMs)在解决高风险的多项选择题测试,在这里代表的考试全国做Ensino Médio(ENEM),多学科的入学考试广泛采用巴西大学的能力。这项考试对LM提出了挑战性的任务,因为它的问题可能涉及多个知识领域,需要理解来自不同领域的信息。例如,一个问题可能需要同时理解统计学和生物学才能解决。这项工作分析了GPT-3.5和GPT-4模型对2009-2017年考试中提出的问题以及2022年考试中的问题产生的响应,这些问题在模型训练完成后公开。此外,不同的提示策略进行了测试,包括使用的思想链(CoT)提示,以产生答案的解释。在2022年版本中,表现最好的模型GPT-4与CoT的准确率达到87%,大大超过GPT-3.5 11分。用于实验的代码和数据可在https://github.com/piresramon/gpt-4-enem上获得。

英文摘要:

The present study aims to explore the capabilities of Language Models (LMs) in tackling high-stakes multiple-choice tests, represented here by the Exame Nacional do Ensino Médio (ENEM), a multidisciplinary entrance examination widely adopted by Brazilian universities. This exam poses challenging tasks for LMs, since its questions may span into multiple fields of knowledge, requiring understanding of information from diverse domains. For instance, a question may require comprehension of both statistics and biology to be solved. This work analyzed responses generated by GPT-3.5 and GPT-4 models for questions presented in the 2009-2017 exams, as well as for questions of the 2022 exam, which were made public after the training of the models was completed. Furthermore, different prompt strategies were tested, including the use of Chain-of-Thought (CoT) prompts to generate explanations for answers. On the 2022 edition, the best-performing model, GPT-4 with CoT, achieved an accuracy of 87%, largely surpassing GPT-3.5 by 11 points. The code and data used on experiments are available at https://github.com/piresramon/gpt-4-enem.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑