SPM-Bench:面向扫描探针显微镜的大型语言模型基准测试
SPM-Bench: Benchmarking Large Language Models for Scanning Probe Microscopy
AI总结:
提出SPM-Bench,一个全自动数据合成管道和严格评估指标(SIP-F1),用于测试LLMs在扫描探针显微镜领域的推理能力,并首次量化模型“个性”。
摘要:
随着LLMs在通用推理方面取得突破,它们在特定科学领域的熟练程度因数据污染、复杂性不足和过高的人力成本而在现有基准测试中暴露出明显差距。在此,我们提出SPM-Bench,一个专为扫描探针显微镜(SPM)设计的原创、博士级多模态基准测试。我们提出一个全自动数据合成管道,确保高权威性和低成本。通过采用锚点门控筛(AGS)技术,我们从2023年至2025年间发表的arXiv和期刊论文中高效提取高价值图像-文本对。通过混合云-本地架构(其中VLM仅返回空间坐标“llbox”以进行本地高保真裁剪),我们的管道在保持高数据集纯度的同时实现了极致的token节省。为了准确客观地评估LLMs的性能,我们引入了严格不完美惩罚F1(SIP-F1)分数。该指标不仅建立了严格的能力层级,而且首次量化了模型“个性”(保守型、激进型、赌徒型或明智型)。通过将这些结果与模型报告的置信度和感知难度相关联,我们揭示了当前AI在复杂物理场景中的真实推理边界。这些见解使SPM-Bench成为自动化科学数据合成的可推广范式。
英文摘要:
As LLMs achieved breakthroughs in general reasoning, their proficiency in specialized scientific domains reveals pronounced gaps in existing benchmarks due to data contamination, insufficient complexity, and prohibitive human labor costs. Here we present SPM-Bench, an original, PhD-level multimodal benchmark specifically designed for scanning probe microscopy (SPM). We propose a fully automated data synthesis pipeline that ensures both high authority and low-cost. By employing Anchor-Gated Sieve (AGS) technology, we efficiently extract high-value image-text pairs from arXiv and journal papers published between 2023 and 2025. Through a hybrid cloud-local architecture where VLMs return only spatial coordinates "llbox" for local high-fidelity cropping, our pipeline achieves extreme token savings while maintaining high dataset purity. To accurately and objectively evaluate the performance of the LLMs, we introduce the Strict Imperfection Penalty F1 (SIP-F1) score. This metric not only establishes a rigorous capability hierarchy but also, for the first time, quantifies model "personalities" (Conservative, Aggressive, Gambler, or Wise). By correlating these results with model-reported confidence and perceived difficulty, we expose the true reasoning boundaries of current AI in complex physical scenarios. These insights establish SPM-Bench as a generalizable paradigm for automated scientific data synthesis.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。