桥接传统可解释性方法与多模态多语言模型：基于XAI的分析
Bridging Traditional Explainability Methods and Multimodal Multilingual Models: An XAI-Based Analysis
AI总结：
提出多模态Shapley值框架，结合频谱图引导的音素对齐（SGPA）预处理方法，实现文本与音频特征的可解释性归因，并开源计算包与可视化工具。
摘要：
多模态大语言模型（MLLMs）有效整合文本和音频以理解复杂交互对话中的上下文。然而，异质模态影响模型行为的内部机制仍然不透明。虽然Shapley值（SV）为基于文本的NLP提供了鲁棒的、模型无关的局部可解释性框架，但其扩展到多模态数据受到跨通道依赖、复杂对话结构以及密集音频表示的高计算复杂性的阻碍。\n在这项工作中，我们形式化了Shapley值框架的多模态扩展，将离散文本标记和对齐的音频片段视为协作特征。为确保计算可行性，我们部署了一套高效的估计策略：低维输入的精确SV计算和基于采样的近似——包括蒙特卡洛排列和具有Neyman最优分配的分层抽样——以在有限计算预算下最小化方差。为解决模态间的粒度不匹配问题，我们提出了频谱图引导的音素对齐（SGPA），一种新颖的预处理方法，将高频音频流映射到可解释的、单词对齐的片段。\n我们的贡献有两方面：首先，我们提供了一个开源的、模型无关的Python包和配套的GUI，用于多模态归因的计算和交互式可视化。其次，我们使用VoiceBench和Infinity Instruct数据集的精选子集，在多种多语言场景下评估我们的框架。实验结果表明，输入模态是归因波动的主要驱动因素，并证明标准句法重要性代理在多模态跨语言上下文中通常无法预测模型注意力。
英文摘要：
Multimodal Large Language Models (MLLMs) effectively integrate text and audio to interpret context in complex interactive dialogues. However, the internal mechanisms by which heterogeneous modalities influence model behavior remain opaque. While Shapley Values (SV) provide a robust, model-agnostic framework for local explainability in text-based NLP, their extension to multimodal data is hindered by cross-channel dependencies, intricate dialogue structures, and the prohibitive computational complexity of dense audio representations. In this work, we formalize a multimodal extension of the Shapley Value framework, treating discrete text tokens and aligned audio segments as cooperative features. To ensure computational feasibility, we deploy a suite of efficient estimation strategies: exact SV computation for low-dimensional inputs and sampling-based approximations - including Monte Carlo permutations and stratified sampling with Neyman-optimal allocation - to minimize variance under constrained computational budgets. To resolve the granularity mismatch between modalities, we propose Spectrogram-Guided Phonetic Alignment (SGPA), a novel preprocessing method that maps high-frequency audio streams to interpretable, word-aligned segments. Our contribution is twofold: first, we provide an open-source, model-agnostic Python package and a companion GUI for the computation and interactive visualization of multimodal attributions. Second, we evaluate our framework using curated subsets of the VoiceBench and Infinity Instruct datasets across diverse multilingual scenarios. Our experimental results reveal that input modality is a primary driver of attribution volatility and demonstrate that standard syntactic importance proxies often fail to predict model attention in multimodal, cross-lingual contexts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
