MIMIC：多模态逆向用于模型解释与概念化
MIMIC: Multimodal Inversion for Model Interpretation and Conceptualization
AI总结：
本文提出MIMIC框架，通过逆向处理多模态模型内部编码，提升模型可解释性与概念化能力，采用联合逆向与特征对齐目标，结合三种正则化器提升空间对齐、图像平滑与语义真实度。
摘要：
视觉语言模型（VLMs）在大规模、复杂且难以解释的架构上编码多模态输入，限制了透明度和信任度。我们提出了一种多模态逆向用于模型解释和概念化（MIMIC）框架，用于逆向VLM的内部编码。MIMIC利用基于VLM的联合逆向和特征对齐目标，以应对VLM的自回归处理。它还包含一个三元组的正则化器，用于空间对齐、自然图像平滑性和语义真实性。我们通过在各种自由形式VLM输出上逆向视觉概念，从定量和定性方面评估MIMIC。报告的结果包括标准视觉质量度量和基于语义的文本度量。据我们所知，这是首个针对VLM概念视觉解释的模型逆向方法。
英文摘要：
Vision Language Models (VLMs) encode multimodal inputs over large, complex, and difficult-to-interpret architectures, which limit transparency and trust. We propose a Multimodal Inversion for Model Interpretation and Conceptualization (MIMIC) framework that inverts the internal encodings of VLMs. MIMIC uses a joint VLM-based inversion and a feature alignment objective to account for VLM's autoregressive processing. It additionally includes a triplet of regularizers for spatial alignment, natural image smoothness, and semantic realism. We evaluate MIMIC both quantitatively and qualitatively by inverting visual concepts across a range of free-form VLM outputs of varying length. Reported results include both standard visual quality metrics and semantic text-based metrics. To the best of our knowledge, this is the first model inversion approach addressing visual interpretations of VLM concepts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
