Many cultural institutions have made large digitized visual collections available online, often under permissible re-use licences. Creating interfaces for exploring and searching these collections is difficult, particularly in the absence of granular metadata. In this paper, we introduce a method for using state-of-the-art multimodal large language models (LLMs) to enable an open-ended, explainable search and discovery interface for visual collections. We show how our approach can create novel clustering and recommendation systems that avoid common pitfalls of methods based directly on visual embeddings. Of particular interest is the ability to offer concrete textual explanations of each recommendation without the need to preselect the features of interest. Together, these features can create a digital interface that is more open-ended and flexible while also being better suited to addressing privacy and ethical concerns. Through a case study using a collection of documentary photographs, we provide several metrics showing the efficacy and possibilities of our approach.
Systems Research Institute of the Polish Academy of Sciences（波兰科学院系统研究所）；Cardinal Stefan Wyszyński University（斯特凡·维辛斯基枢机大学）
In this article we present the effects of our work on the subject of the technical approach to the 3D point cloud data analysis through the use of the Octree method to compress, analyse and compute the initial data.
Comparison of Different Deep Neural Network Models in the Cultural Heritage Domain
Institute of Mathematics and Informatics, Bulgarian Academy of Sciences（数学与信息学研究所，保加利亚科学院）；Institute of Information Science and Technologies, Consiglio Nazionale delle Ricerche（信息科学与技术研究所，意大利国家研究理事会）
Study of cultural-heritage objects with embellished realistic and abstract designs made up of connected and intertwined curves crosscuts a number of related disciplines, including archaeology, art history, and heritage management. However, many objects, such as pottery sherds found in the archaeological record, are fragmentary, making the underlying complete designs unknowable at the scale of the sherd fragment. The challenge to reconstruct and study complete designs is stymied because 1) most fragmentary cultural-heritage objects contain only a small portion of the underlying full design, 2) …
Advancements in optical metrology has enabled documentation of dense 3D point clouds of cultural heritage sites. For large scale and continuous digital documentation, processing of dense 3D point clouds becomes computationally cumbersome, and often requires additional hardware for data management, increasing the time cost, and complexity of projects. To this end, this manuscript presents an original approach to generate fast and reliable semantic digital models of heritage hemispherical domes using only two images. New closed formulations were derived to establish the relationships between sph…
Cyprus University of Technology（塞浦路斯理工大学）；National Technical University of Athens（雅典国家技术大学）
Underwater Cultural Heritage (CH) sites are widely spread; from ruins in coastlines up to shipwrecks in deep. The documentation and preservation of this heritage is an obligation of the mankind, dictated also by the international treaties like the Convention on the Protection of the Underwater Cultural Her-itage which fosters the use of "non-destructive techniques and survey meth-ods in preference over the recovery of objects". However, submerged CH lacks in protection and monitoring in regards to the land CH and nowadays recording and documenting, for digital preservation as well as dissemina…
University of South Carolina（南卡罗来纳大学）；South Carolina Institute of Archaeology and Anthropology, University of South Carolina（南卡罗来纳大学南卡罗来纳考古与人类学研究所）；Center for Digital Humanities, University of South Carolina（南卡罗来纳大学数字人文中心）；Tianjin University（天津大学）
Motivated by the important archaeological application of exploring cultural heritage objects, in this paper we study the challenging problem of automatically segmenting curve structures that are very weakly stamped or carved on an object surface in the form of a highly noisy depth map. Different from most classical low-level image segmentation methods that are known to be very sensitive to the noise and occlusions, we propose a new supervised learning algorithm based on Convolutional Neural Network (CNN) to implicitly learn and utilize more curve geometry and pattern information for addressing…
In this paper we present a semi-automatic 2D-3D local registration pipeline capable of coloring 3D models obtained from 3D scanners by using uncalibrated images. The proposed pipeline exploits the Structure from Motion (SfM) technique in order to reconstruct a sparse representation of the 3D object and obtain the camera parameters from image feature matches. We then coarsely register the reconstructed 3D model to the scanned one through the Scale Iterative Closest Point (SICP) algorithm. SICP provides the global scale, rotation and translation parameters, using minimal manual user intervention…
ChinaHeritaQA: A Culturally-Grounded Visual Question Answering Dataset for World Heritage Sites in China
LMU Munich（慕尼黑大学）；FAU Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；University of Tübingen & Tübingen AI Center（图宾根大学与图宾根人工智能中心）；Sun Yat-sen University（中山大学）；University of Copenhagen（哥本哈根大学）；University of Maryland, College Park（马里兰大学帕克分校）
Augmented Reality in Cultural Heritage: A Dual-Model Pipeline for 3D Artwork Reconstruction
Department of Computer Science, Sapienza University（计算机科学系，萨皮恩扎大学）；Department of Mathematics and Computer Science, University of Udine（数学与计算机科学系，乌迪内大学）
AmalthAI: An Open-Source Computer Vision Platform for Cultural Heritage
Democritus University of Thrace（德谟克利特色雷斯大学）；Athena Research Center（雅典娜研究中心）；National and Kapodistrian University of Athens（雅典国立卡波迪斯特里亚大学）；University of Warsaw（华沙大学）
Solving Jigsaw Puzzles in the Wild: Human-Guided Reconstruction of Cultural Heritage Fragments
DAIS, Ca’ Foscari University of Venice（威尼斯Ca’ Foscari大学DAIS部门）；Department of Historical Studies, University of Milan（米兰大学历史研究系）；Zhejiang Normal University（浙江师范大学）
High Resolution Surface Reconstruction of Cultural Heritage Objects Using Shape from Polarization Method
Institute of Cartography and Geoinformatics, Leibniz University Hannover（莱布尼茨汉诺威大学制图与地理信息研究所）；School of Surveying and Geospatial Engineering, College of Engineering, University of Tehran（德黑兰大学工程学院测绘与地理空间工程系）
An Inclusive and Lightweight Approach to Federated Continual Learning for Cultural Heritage
Information Technologies Institute, Centre for Research and Technology Hellas（希腊研究与技术中心信息技术研究所）；School of Electronic Engineering and Computer Science, Queen Mary University of London（伦敦大学玛丽女王学院电子工程与计算机科学学院）
National Technological University of Athens（雅典国家技术大学）
In this paper, we scrutinize the effectiveness of various clustering techniques, investigating their applicability in Cultural Heritage monitoring applications. In the context of this paper, we detect the level of decomposition and corrosion on the walls of Saint Nicholas fort in Rhodes utilizing hyperspectral images. A total of 6 different clustering approaches have been evaluated over a set of 14 different orthorectified hyperspectral images. Experimental setup in this study involves K-means, Spectral, Meanshift, DBSCAN, Birch and Optics algorithms. For each of these techniques we evaluate i…
This work presents the development of a flexible microscopic chemical imaging platform for cultural heritage that utilizes wavelength-tunable oblique illumination from a point source to obtain per-pixel reflectance spectra in the VIS-NIR range. The microscope light source can be adjusted on two axes allowing for a hemisphere of possible illumination directions. The synthesis of multiple illumination angles allows for the calculation of surface normal vectors, similar to phase gradients, and axial optical sectioning. The extraction of spectral reflectance images with high spatial resolutions fr…
National Technical University of Athens（雅典国立技术大学）；University of Malta（马耳他大学）
In Cultural Heritage, hyperspectral images are commonly used since they provide extended information regarding the optical properties of materials. Thus, the processing of such high-dimensional data becomes challenging from the perspective of machine learning techniques to be applied. In this paper, we propose a Rank-$R$ tensor-based learning model to identify and classify material defects on Cultural Heritage monuments. In contrast to conventional deep learning approaches, the proposed high order tensor-based learning demonstrates greater accuracy and robustness against overfitting. Experimen…
Reviving Cultural Heritage: A Novel Approach for Comprehensive Historical Document Restoration
South China University of Technology（南方科技大学）；Intsig Information Co., Ltd.（Intsig信息有限公司）；INTSIG-SCUT Joint Lab on Document Analysis and Recognition（INTSIG-SCUT文档分析与识别联合实验室）；SCUT-Zhuhai Institute of Modern Industrial Innovation（南方科技大学珠海现代工业创新研究院）
Media Integration and Communication Center (MICC)（媒体集成与通信中心（MICC））；University of Florence（佛罗伦萨大学）
Technology and the fruition of cultural heritage are becoming increasingly more entwined, especially with the advent of smart audio guides, virtual and augmented reality, and interactive installations. Machine learning and computer vision are important components of this ongoing integration, enabling new interaction modalities between user and museum. Nonetheless, the most frequent way of interacting with paintings and statues still remains taking pictures. Yet images alone can only convey the aesthetics of the artwork, lacking is information which is often required to fully understand and app…
Kubelka-Munk (K-M) theory has been successfully used to estimate pigment concentrations in the pigment mixtures of modern paintings in spectral imagery. In this study the single-constant K-M theory has been utilized for the classification of green pigments in the Selden Map of China, a navigational map of the South China Sea likely created in the early seventeenth century. Hyperspectral data of the map was collected at the Bodleian Library, University of Oxford, and can be used to estimate the pigment diversity, and spatial distribution, within the map. This work seeks to assess the utility of…
VaseVQA: Multimodal Agent and Benchmark for Ancient Greek Pottery
University of Adelaide（阿德莱德大学）；University of Liverpool（利物浦大学）；University of Technology Sydney（悉尼技术大学）；The Australian National University（澳大利亚国立大学）；La Trobe University（拉特罗布大学）
ENSTA, Institut Polytechnique Paris（巴黎综合理工学院ENSTA）；Inria（法国国家信息与自动化研究所）；DaSCI Andalusian Institute in Data Science and Computational Intelligence, University of Granada（格拉纳达大学DaSCI安达卢西亚数据科学与计算智能研究所）；University of Granada（格拉纳达大学）；Free University of Bozen-Bolzano（波尔扎诺自由大学）
The latest Deep Learning (DL) models for detection and classification have achieved an unprecedented performance over classical machine learning algorithms. However, DL models are black-box methods hard to debug, interpret, and certify. DL alone cannot provide explanations that can be validated by a non technical audience. In contrast, symbolic AI systems that convert concepts into rules or symbols -- such as knowledge graphs -- are easier to explain. However, they present lower generalisation and scaling capabilities. A very important challenge is to fuse DL representations with expert knowle…
