Heri-Graphs：一个为多模态机器学习创建数据集的工作流程，用于使用社交媒体对遗产价值和属性的图形进行学习
Heri-Graphs: A Workflow of Creating Datasets for Multi-modal Machine Learning on Graphs of Heritage Values and Attributes with Social Media
摘要：
价值（为什么要保护）和属性（保护什么）是文化遗产的基本概念。最近的研究一直在使用社交媒体来绘制公众传达给文化遗产的价值观和属性。然而，它是罕见的连接异构形式的图像，文本，地理位置，时间戳和社会网络结构，挖掘其中的语义和结构特征。这项研究提出了一种方法的工作流程，用于构建这样的多模态数据集，使用Flickr上的帖子和图像，用于基于图形的机器学习（ML）任务，有关遗产的价值和属性。在使用最先进的ML模型进行数据预处理后，视觉内容和文本语义的多模态信息被建模为节点特征和标签，而它们的社会关系和时空上下文被建模为Multi-Graphs中的链接。该工作流程在三个包含联合国教科文组织世界遗产的城市-阿姆斯特丹，苏州和威尼斯进行了测试，为半监督学习任务提供了具有高度一致性的数据集。整个过程用数学符号进行了正式描述，可以应用于临时任务中，既可以作为具有技术相关性的ML问题，也可以作为具有社会利益的城市/遗产研究问题。这项研究还有助于了解和绘制遗产价值和属性，以便今后在全球案例中进行研究，旨在采取包容性遗产管理做法。
英文摘要：
Values (why to conserve) and Attributes (what to conserve) are essential concepts of cultural heritage. Recent studies have been using social media to map values and attributes conveyed by public to cultural heritage. However, it is rare to connect heterogeneous modalities of images, texts, geo-locations, timestamps, and social network structures to mine the semantic and structural characteristics therein. This study presents a methodological workflow for constructing such multi-modal datasets using posts and images on Flickr for graph-based machine learning (ML) tasks concerning heritage values and attributes. After data pre-processing using state-of-the-art ML models, the multi-modal information of visual contents and textual semantics are modelled as node features and labels, while their social relationships and spatiotemporal contexts are modelled as links in Multi-Graphs. The workflow is tested in three cities containing UNESCO World Heritage properties - Amsterdam, Suzhou, and Venice, which yielded datasets with high consistency for semi-supervised learning tasks. The entire process is formally described with mathematical notations, ready to be applied in provisional tasks both as ML problems with technical relevance and as urban/heritage study questions with societal interests. This study could also benefit the understanding and mapping of heritage values and attributes for future research in global cases, aiming at inclusive heritage management practices.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
