在巴西国家标准考试框架下考察 LLM 架构的行为
Examining the Behavior of LLM Architectures Within the Framework of Standardized National Exams in Brazil
AI总结：
本文利用 ENEM 公开答题与社会经济数据，比较 GPT-3.5、GPT-4、MariTalk 与不同 SES 学生在客观题和作文上的表现，揭示模型答案不代表任何人类群体。
摘要：
Exame Nacional do Ensino Médio（ENEM，巴西全国中学教育考试）是巴西学生的一项关键考试，是进入巴西大量大学所必需的。该考试由数学、人文、自然科学和语言四门高中水平的客观题考试以及一篇写作作文组成。由于巴西政府的透明度政策，学生对考试及随附社会经济地位问卷的回答每年都会公开（尽管经过匿名化处理）。在大型语言模型（LLM）的背景下，这些数据非常适合用来比较不同人类群体与 AI，因为我们可以获得人类和机器的答案分布。我们利用 ENEM 数据集的这些特征，将 GPT-3.5、GPT-4 以及使用葡萄牙语数据训练的模型 MariTalk 与人类进行比较，旨在确定它们的答案如何与真实社会群体相关联，以及这可能揭示模型的哪些偏见。我们按照社会经济地位（SES）划分人类群体，并针对每一道题目和作文，将他们的答案分布与 LLM 进行比较。在多项选择式巴西葡萄牙语考试中比较 LLM 与人类的表现时，我们没有发现显著偏见，因为模型与人类答案之间的距离主要由人类的准确率决定。在考察生成文本时也得出了类似结论：分析作文时，我们观察到人类作文与 LLM 作文在若干关键因素上存在差异，其中之一是用词选择，模型作文能够很容易地与人类作文区分开。文本在句法上也存在差异，除其他差异外，LLM 生成的作文平均表现出更短的句子和更少的思想单元。这些结果表明，在 ENEM 语境下的巴西葡萄牙语中，LLM 的输出并不代表任何人类群体，并且在所有考试中都与巴西学生的答案存在显著差异。
英文摘要：
The Exame Nacional do Ensino Médio (ENEM) is a pivotal test for Brazilian students, required for admission to a significant number of universities in Brazil. The test consists of four objective high-school level tests on Math, Humanities, Natural Sciences and Languages, and one writing essay. Students' answers to the test and to the accompanying socioeconomic status questionnaire are made public every year (albeit anonymized) due to transparency policies from the Brazilian Government. In the context of large language models (LLMs), these data lend themselves nicely to comparing different groups of humans with AI, as we can have access to human and machine answer distributions. We leverage these characteristics of the ENEM dataset and compare GPT-3.5 and 4, and MariTalk, a model trained using Portuguese data, to humans, aiming to ascertain how their answers relate to real societal groups and what that may reveal about the model biases. We divide the human groups by using socioeconomic status (SES), and compare their answer distribution with LLMs for each question and for the essay. We find no significant biases when comparing LLM performance to humans on the multiple-choice Brazilian Portuguese tests, as the distance between model and human answers is mostly determined by the human accuracy. A similar conclusion is found by looking at the generated text as, when analyzing the essays, we observe that human and LLM essays differ in a few key factors, one being the choice of words where model essays were easily separable from human ones. The texts also differ syntactically, with LLM generated essays exhibiting, on average, smaller sentences and less thought units, among other differences. These results suggest that, for Brazilian Portuguese in the ENEM context, LLM outputs represent no group of humans, being significantly different from the answers from Brazilian students across all tests.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
