We introduce a new challenge to test the STEM skills of neural models. The problems in the real world often require solutions, combining knowledge from STEM (science, technology, engineering, and math). Unlike existing datasets, our dataset requires the understanding of multimodal vision-language information of STEM. Our dataset features one of the largest and most comprehensive datasets for the challenge. It includes 448 skills and 1,073,146 questions spanning all STEM subjects. Compared to existing datasets that often focus on examining expert-level ability, our dataset includes fundamental skills and questions designed based on the K-12 curriculum. We also add state-of-the-art foundation models such as CLIP and GPT-3.5-Turbo to our benchmark. Results show that the recent model advances only help master a very limited number of lower grade-level skills (2.5% in the third grade) in our dataset. In fact, these models are still well below (averaging 54.7%) the performance of elementary students, not to mention near expert-level performance. To understand and increase the performance on our dataset, we teach the models on a training split of our dataset. Even though we observe improved performance, the model performance remains relatively low compared to average elementary students. To solve STEM problems, we will need novel algorithmic innovations from the community.
PortexAI;University of Milano-Bicocca(米兰-比科卡大学);University of Calgary(卡尔加里大学);University of Chicago(芝加哥大学);Inria(法国国家信息与自动化技术研究院);Fraunhofer Institute for Individualized Medical Technology IMTE(弗劳恩霍夫个性化医疗技术研究所);University of Cambridge(剑桥大学);Independent(独立);McGill University(麦吉尔大学);Massachusetts Institute of Technology(麻省理工学院);École Polytechnique Fédérale de Lausanne(洛桑联邦理工学院);Queen Mary University of London(伦敦大学玛丽女王学院);Dot Ingredients;National University of Singapore(新加坡国立大学);Georgia Institute of Technology(佐治亚理工学院);University of Edinburgh(爱丁堡大学);Murdoch University(墨尔本大学);University of Porto(波尔图大学);Stanford University(斯坦福大学);Stony Brook University(史泰津布鲁克大学)
Toward LLM-Supported Automated Assessment of Critical Thinking Subskills
University of Minnesota(明尼苏达大学);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);Adelaide University(阿德莱德大学);Brighter Research(Brighter研究公司);Educational Testing Service (ETS)(教育考试服务中心(ETS))
Riveter: Measuring Power and Social Dynamics Between Entities
Allen Institute for AI(艾伦人工智能研究所);Johns Hopkins University(约翰斯·霍普金斯大学);Carnegie Mellon University(卡内基梅隆大学);University of Washington(华盛顿大学);Emory University(埃默里大学)