SoM-1K:用于材料力学能力的千题基准数据集
SoM-1K: A Thousand-Problem Benchmark Dataset for Strength of Materials
AI总结:
该研究推出含1065道题的SoM-1K多模态基准,评估8种基础模型的材料力学能力,提出DoI提示策略,发现当前模型表现差,LLMs配DoI优于VLMs配图像,为工程AI提供基准并强调多模态推理需求。
摘要:
基础模型在多个领域展现出出色能力,但在复杂多模态工程问题上的表现仍未得到充分探索。我们推出SoM-1K,这是首个用于评估基础模型材料力学(Strength of Materials, SoM)问题能力的大规模多模态基准数据集。该数据集包含1065道带标注的材料力学问题,通过同时包含文本问题陈述和示意图,模拟真实工程任务。由于当前基础模型理解复杂视觉信息的能力有限,我们提出一种名为图像描述(Descriptions of Images, DoI)的新型提示策略,该策略提供由专家生成的严格文本描述作为视觉图的上下文。我们评估了8种代表性基础模型,包括大语言模型(Large Language Models, LLMs)和视觉语言模型(Vision Language Models, VLMs)。结果显示,当前基础模型在这些工程问题上表现显著不佳,性能最佳的模型仅达到56.6%的准确率。有趣的是,我们发现当提供DoI时,LLMs的表现往往优于提供视觉图的VLMs。详细的错误分析表明,DoI在减轻视觉误解错误方面发挥关键作用,这表明对于当前基础模型而言,准确的基于文本的描述可能比直接图像输入更有效。本研究为工程AI建立了严格的基准,并强调开发基础模型更强大的多模态推理能力的迫切需求,尤其是在科学和工程领域。
英文摘要:
Foundation models have shown remarkable capabilities in various domains, but their performance on complex, multimodal engineering problems remains largely unexplored. We introduce SoM-1K, the first large-scale multimodal benchmark dataset dedicated to evaluating foundation models on problems in the strength of materials (SoM). The dataset, which contains 1,065 annotated SoM problems, mirrors real-world engineering tasks by including both textual problem statements and schematic diagrams. Due to the limited capabilities of current foundation models in understanding complicated visual information, we propose a novel prompting strategy called Descriptions of Images (DoI), which provides rigorous expert-generated text descriptions of the visual diagrams as the context. We evaluate eight representative foundation models, including both large language models (LLMs) and vision language models (VLMs). Our results show that current foundation models struggle significantly with these engineering problems, with the best-performing model achieving only 56.6% accuracy. Interestingly, we found that LLMs, when provided with DoI, often outperform VLMs provided with visual diagrams. A detailed error analysis reveals that DoI plays a crucial role in mitigating visual misinterpretation errors, suggesting that accurate text-based descriptions can be more effective than direct image input for current foundation models. This work establishes a rigorous benchmark for engineering AI and highlights a critical need for developing more robust multimodal reasoning capabilities in foundation models, particularly in scientific and engineering contexts.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。