arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.17205cs.CLcs.AIcs.LG
收藏

测量神经模型的视觉-语言STEM技能

Measuring Vision-Language STEM Skills of Neural Models

School of Computer Science, Peking University(北京大学计算机学院);Washington University in St. Louis(圣路易斯华盛顿大学)

Jianhao Shen, Ye Yuan, Srbuhi Mirzoyan, Ming Zhang, Chenguang Wang

AI总结:

本文提出一个涵盖448项技能和107万道题的多模态STEM测试基准,发现CLIP和GPT-3.5等先进模型仅掌握少量低年级技能,性能远低于小学生,呼吁算法创新。

摘要:

我们引入了一个新的挑战来测试神经模型的STEM技能。现实世界中的问题往往需要结合STEM(科学、技术、工程和数学)的知识来解决。与现有数据集不同,我们的数据集要求理解STEM的多模态视觉-语言信息。我们的数据集是该挑战中最大、最全面的数据集之一。它包含448项技能和1,073,146个问题,涵盖所有STEM学科。与通常侧重于考察专家级能力的现有数据集相比,我们的数据集包含基于K-12课程设计的基础技能和问题。我们还将CLIP和GPT-3.5-Turbo等最先进的基础模型加入我们的基准测试。结果表明,最近的模型进展仅能掌握我们数据集中非常有限的低年级技能(三年级中占2.5%)。事实上,这些模型的性能(平均54.7%)仍远低于小学生的水平,更不用说接近专家级水平了。为了理解并提高我们数据集上的性能,我们在数据集的训练子集上对模型进行训练。尽管我们观察到性能有所提升,但模型性能相对于普通小学生仍相对较低。要解决STEM问题,我们需要社区提出新颖的算法创新。

英文摘要:

We introduce a new challenge to test the STEM skills of neural models. The problems in the real world often require solutions, combining knowledge from STEM (science, technology, engineering, and math). Unlike existing datasets, our dataset requires the understanding of multimodal vision-language information of STEM. Our dataset features one of the largest and most comprehensive datasets for the challenge. It includes 448 skills and 1,073,146 questions spanning all STEM subjects. Compared to existing datasets that often focus on examining expert-level ability, our dataset includes fundamental skills and questions designed based on the K-12 curriculum. We also add state-of-the-art foundation models such as CLIP and GPT-3.5-Turbo to our benchmark. Results show that the recent model advances only help master a very limited number of lower grade-level skills (2.5% in the third grade) in our dataset. In fact, these models are still well below (averaging 54.7%) the performance of elementary students, not to mention near expert-level performance. To understand and increase the performance on our dataset, we teach the models on a training split of our dataset. Even though we observe improved performance, the model performance remains relatively low compared to average elementary students. To solve STEM problems, we will need novel algorithmic innovations from the community.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑