当图遇见多模态：多模态属性图学习的基准测试与思考
When Graph meets Multimodal: Benchmarking and Meditating on Multimodal Attributed Graphs Learning
AI总结：
本文提出MAGB基准数据集，系统评估GNN与VLM在多模态属性图学习中的表现，揭示模态重要性差异及多模态嵌入的效用，为MAG研究提供可靠基准。
摘要：
多模态属性图（MAGs）在现实应用中无处不在，通过节点上附加的多模态属性（如文本和图像）以及表示节点交互的拓扑结构，涵盖了广泛的知识。尽管其有潜力推动社交网络和电子商务等多个研究领域的发展，但由于缺乏标准化的数据集和评估框架，MAG表示学习（MAGRL）仍未被充分探索。在本文中，我们首先提出了MAGB，一个全面的MAG基准数据集，包含来自不同领域的、具有文本和视觉属性的精选图。基于MAGB数据集，我们进一步系统地评估了两种主流的MAGRL范式：$\ extit{GNN-as-Predictor}$，它通过图神经网络（GNNs）整合多模态属性，以及$\ extit{VLM-as-Predictor}$，它利用视觉语言模型（VLMs）进行零样本推理。在MAGB上进行的大量实验揭示了以下关键见解：$\ extit{(i)}$ 模态重要性随特定领域特征而剧烈波动。$\ extit{(ii)}$ 多模态嵌入可以提升GNNs的性能上限。然而，模态之间的内在偏差可能阻碍有效训练，尤其是在低数据场景下。$\ extit{(iii)}$ VLMs在生成多模态嵌入以缓解文本和视觉属性之间的不平衡方面非常有效。这些发现阐明了多模态属性与图拓扑之间的协同作用，有助于建立可靠的基准，为未来MAG研究铺平道路。MAGB数据集和评估流程公开于https://github.com/sktsherlock/MAGB。
英文摘要：
Multimodal Attributed Graphs (MAGs) are ubiquitous in real-world applications, encompassing extensive knowledge through multimodal attributes attached to nodes (e.g., texts and images) and topological structure representing node interactions. Despite its potential to advance diverse research fields like social networks and e-commerce, MAG representation learning (MAGRL) remains underexplored due to the lack of standardized datasets and evaluation frameworks. In this paper, we first propose MAGB, a comprehensive MAG benchmark dataset, featuring curated graphs from various domains with both textual and visual attributes. Based on MAGB dataset, we further systematically evaluate two mainstream MAGRL paradigms: $\textit{GNN-as-Predictor}$, which integrates multimodal attributes via Graph Neural Networks (GNNs), and $\textit{VLM-as-Predictor}$, which harnesses Vision Language Models (VLMs) for zero-shot reasoning. Extensive experiments on MAGB reveal following critical insights: $\textit{(i)}$ Modality significances fluctuate drastically with specific domain characteristics. $\textit{(ii)}$ Multimodal embeddings can elevate the performance ceiling of GNNs. However, intrinsic biases among modalities may impede effective training, particularly in low-data scenarios. $\textit{(iii)}$ VLMs are highly effective at generating multimodal embeddings that alleviate the imbalance between textual and visual attributes. These discoveries, which illuminate the synergy between multimodal attributes and graph topologies, contribute to reliable benchmarks, paving the way for future MAG research. The MAGB dataset and evaluation pipeline are publicly available at https://github.com/sktsherlock/MAGB.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
