PictOBI-20k：揭示大型多模态模型在象形甲骨文视觉解读中的能力
PictOBI-20k: Unveiling Large Multimodal Models in Visual Decipherment for Pictographic Oracle Bone Characters
AI总结：
本文提出PictOBI-20k数据集，含20k甲骨文与实物图像及15k多选问题，评估大型多模态模型在象形甲骨文视觉解读中的能力，发现模型受语言先验限制，未充分利用视觉信息。
摘要：
解读甲骨文（OBC）作为已知最早的中文书写形式，一直是学者们终极且坚定不移的目标，为理解人类早期生产方式提供了不可替代的关键。当前甲骨文的解读方法主要受限于考古发掘的零散性和铭文语料库的有限性。借助大型多模态模型（LMMs）强大的视觉感知能力，利用LMMs进行甲骨文视觉解读的潜力日益增加。在本文中，我们引入了PictOBI-20k数据集，旨在评估LMMs在象形甲骨文视觉解读任务上的表现。该数据集包含精心收集的20k张甲骨文和真实物体图像，构成了超过15k道多项选择题。我们还进行了主观标注，以调查人类与LMMs在视觉推理中参考点的一致性。实验表明，通用LMMs具备初步的视觉解读技能，但LMMs并未有效利用视觉信息，大多数时候受限于语言先验。我们希望我们的数据集能够促进未来面向甲骨文的LMMs在视觉注意力方面的评估和优化。代码和数据集将在https://github.com/OBI-Future/PictOBI-20k提供。
英文摘要：
Deciphering oracle bone characters (OBCs), the oldest attested form of written Chinese, has remained the ultimate, unwavering goal of scholars, offering an irreplaceable key to understanding humanity's early modes of production. Current decipherment methodologies of OBC are primarily constrained by the sporadic nature of archaeological excavations and the limited corpus of inscriptions. With the powerful visual perception capability of large multimodal models (LMMs), the potential of using LMMs for visually deciphering OBCs has increased. In this paper, we introduce PictOBI-20k, a dataset designed to evaluate LMMs on the visual decipherment tasks of pictographic OBCs. It includes 20k meticulously collected OBC and real object images, forming over 15k multi-choice questions. We also conduct subjective annotations to investigate the consistency of the reference point between humans and LMMs in visual reasoning. Experiments indicate that general LMMs possess preliminary visual decipherment skills, and LMMs are not effectively using visual information, while most of the time they are limited by language priors. We hope that our dataset can facilitate the evaluation and optimization of visual attention in future OBC-oriented LMMs. The code and dataset will be available at https://github.com/OBI-Future/PictOBI-20k.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
