arXiv 2603.267982026-08-11cs.LGcs.AI

解释、验证和对齐视觉语言模型嵌入中的语义层次结构

Explaining, Verifying, and Aligning Semantic Hierarchies in Vision-Language Model Embeddings

University of Lübeck（吕贝克大学）；Technical University of Munich（慕尼黑工业大学）；AUMOVIO SE；Osnabrück University（奥斯纳布吕克大学）；Leibniz Institute for Agricultural Engineering and Bioeconomy（莱布尼茨农业工程与生物经济研究所）；University of Hamburg（汉堡大学）

Gesina Schwalbe, Mert Keser, Moritz Bayerkuhnlein, Edgar Heinert, Annika Mütze, Marvin Keller, Sparsh Tiwari, Georgii Mikriukov, Diedrich Wolter, Jae Hee Lee, Matthias Rottmann

备注：: camera-ready version of accepted IJCAI-ECAI 2026 paper including supplementary material; code: https://github.com/gesina/ontological-commitment

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出一种框架，用于解释、验证和对齐视觉语言模型嵌入中的语义层次结构，通过聚类和概念库匹配提取层次结构，并利用人类本体评估其合理性，最终提出基于本体的对齐方法以提升共享嵌入空间的语义对齐。

摘要：

视觉语言模型（VLM）编码器如CLIP能够在共享的图像-文本嵌入空间中实现强大的检索和零样本分类，但该空间的语义组织 rarely 被 inspect。我们提出一种 post-hoc 框架，用于解释、验证和对齐由 VLM 在给定子类集上诱导的语义层次结构。首先，我们通过聚类类中心提取二元层次结构，并通过字典匹配将内部节点命名到概念库。其次，我们通过高效树和边级一致性度量将提取的树与人类本体进行比较以量化合理性，并通过可解释的层次树遍历推理和不确定性感知的早停（UAES）评估实用性。第三，我们提出一种基于本体的 post-hoc 对齐方法，通过 UMAP 从期望的层次结构生成目标邻域，学习轻量级嵌入空间转换。在13个预训练 VLM 和4个图像数据集中，我们的方法发现系统性的模态差异：图像编码器更具判别性，而文本编码器诱导的层次结构更符合人类分类法。总体而言，结果揭示了零样本准确率与本体合理性之间的持久权衡，并提出了改进共享嵌入空间语义对齐的实用途径。

英文摘要：

Vision-language model (VLM) encoders such as CLIP enable strong retrieval and zero-shot classification in a shared image-text embedding space, yet the semantic organization of this space is rarely inspected. We present a post-hoc framework to explain, verify, and align the semantic hierarchies induced by a VLM over a given set of child classes. First, we extract a binary hierarchy by agglomerative clustering of class centroids and name internal nodes by dictionary-based matching to a concept bank. Second, we quantify plausibility by comparing the extracted tree against human ontologies using efficient tree- and edge-level consistency measures, and we evaluate utility via explainable hierarchical tree-traversal inference with uncertainty-aware early stopping (UAES). Third, we propose an ontology-guided post-hoc alignment method that learns a lightweight embedding-space transformation, using UMAP to generate target neighborhoods from a desired hierarchy. Across 13 pretrained VLMs and 4 image datasets, our method finds systematic modality differences: image encoders are more discriminative, while text encoders induce hierarchies that better match human taxonomies. Overall, the results reveal a persistent trade-off between zero-shot accuracy and ontological plausibility and suggest practical routes to improve semantic alignment in shared embedding spaces.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。