arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.01190cs.IRcs.DL
收藏

基于CLIP的大比例尺地图集检索

Integrating Visual and Textual Inputs for Searching Large-Scale Map Collections with CLIP

Jamie Mahowald, Benjamin Charles Germain Lee

摘要:

尽管地图在数字收藏中的流行和历史重要性,但当前导航和探索地图收藏的方法在很大程度上限于目录记录和结构化元数据。在本文中,我们探讨了使用自然语言输入(“海怪地图”)、视觉输入(即,反向图像搜索)和多模式输入(示例地图+“更多灰度”)。作为一个案例研究,我们采用了562,842张地图图像,这些图像可以通过美国国会图书馆的API公开访问。为了实现这一点,我们使用多模态对比图像预训练(CLIP)机器学习模型来生成这些地图的嵌入,并开发代码来实现这些输入策略的探索性搜索功能。我们目前的结果,例如搜索创建在与美国国会图书馆的地理和地图司的工作人员协商,并描述这些搜索查询的优势,劣势和可能性。此外,我们还介绍了一个包含10,504个地图标题对的微调数据集,以及一个在该数据集上微调CLIP模型的架构。为了便于重复使用,我们将所有代码都提供在文档化的交互式笔记本中,并将所有代码置于公共领域。最后,我们讨论了在画廊,图书馆,档案馆和博物馆举办的数字化和非数字化收藏中应用这些方法的机遇和挑战。

英文摘要:

Despite the prevalence and historical importance of maps in digital collections, current methods of navigating and exploring map collections are largely restricted to catalog records and structured metadata. In this paper, we explore the potential for interactively searching large-scale map collections using natural language inputs ("maps with sea monsters"), visual inputs (i.e., reverse image search), and multimodal inputs (an example map + "more grayscale"). As a case study, we adopt 562,842 images of maps publicly accessible via the Library of Congress's API. To accomplish this, we use the mulitmodal Contrastive Language-Image Pre-training (CLIP) machine learning model to generate embeddings for these maps, and we develop code to implement exploratory search capabilities with these input strategies. We present results for example searches created in consultation with staff in the Library of Congress's Geography and Map Division and describe the strengths, weaknesses, and possibilities for these search queries. Moreover, we introduce a fine-tuning dataset of 10,504 map-caption pairs, along with an architecture for fine-tuning a CLIP model on this dataset. To facilitate re-use, we provide all of our code in documented, interactive Jupyter notebooks and place all code into the public domain. Lastly, we discuss the opportunities and challenges for applying these approaches across both digitized and born-digital collections held by galleries, libraries, archives, and museums.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑