评估 ChatGPT-4 Vision 在巴西全国计算机科学本科考试中的表现
Evaluating ChatGPT-4 Vision on Brazil's National Undergraduate Computer Science Exam
AI总结：
本文以巴西2021年 ENADE 计算机科学试题评估 ChatGPT-4 Vision，发现其成绩进入前10%，但仍需人工监督并暴露试题设计问题。
摘要：
近期，视觉能力被集成到大语言模型（LLM）中，这有望在科技教育中发挥关键作用，因为在科技教育中，图示、图表和表格等视觉元素通常被用于改善学习体验。本研究考察了 ChatGPT-4 Vision 在巴西2021年全国本科考试（ENADE）计算机科学学士学位部分中的表现；在研究开展时，该模型是 OpenAI 最先进的视觉模型。我们以原始图像形式向该模型呈现考试中的开放性试题和多项选择题，并允许其针对不同答案键进行重新评估，从而能够在一项涉及文本和视觉内容的大规模学业评估中评价模型的推理和自我反思能力。ChatGPT-4 Vision 的表现显著优于考试参与者的平均水平，成绩位于前10%的最佳分数百分位。尽管它在包含视觉元素的试题中表现出色，但在题意理解、逻辑推理和视觉敏锐度方面也遇到了挑战。一个独立专家小组参与审查模型答案与答案键不一致的案例，结果揭示了一些含有模糊或歧义表述的拙劣试题，这促使人们关注未来考试中改进试题设计的关键必要性。我们的研究结果表明，尽管 ChatGPT-4 Vision 在多模态学业评估中展现出前景，但人工监督对于核验模型准确性并确保高利害教育考试的公平性仍然至关重要。本文的研究材料可在 https://github.com/nabormendonca/gpt-4v-enade-cs-2021 公开获取。
英文摘要：
The recent integration of visual capabilities into Large Language Models (LLMs) has the potential to play a pivotal role in science and technology education, where visual elements such as diagrams, charts, and tables are commonly used to improve the learning experience. This study investigates the performance of ChatGPT-4 Vision, OpenAI's most advanced visual model at the time the study was conducted, on the Bachelor in Computer Science section of Brazil's 2021 National Undergraduate Exam (ENADE). By presenting the model with the exam's open and multiple-choice questions in their original image format and allowing for reassessment in response to differing answer keys, we were able to evaluate the model's reasoning and self-reflecting capabilities in a large-scale academic assessment involving textual and visual content. ChatGPT-4 Vision significantly outperformed the average exam participant, positioning itself within the top 10 best score percentile. While it excelled in questions that incorporated visual elements, it also encountered challenges with question interpretation, logical reasoning, and visual acuity. The involvement of an independent expert panel to review cases of disagreement between the model and the answer key revealed some poorly constructed questions containing vague or ambiguous statements, calling attention to the critical need for improved question design in future exams. Our findings suggest that while ChatGPT-4 Vision shows promise in multimodal academic evaluations, human oversight remains crucial for verifying the model's accuracy and ensuring the fairness of high-stakes educational exams. The paper's research materials are publicly available at https://github.com/nabormendonca/gpt-4v-enade-cs-2021.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
