基于影响估计的大语言模型数据描述生成
Data Descriptions from Large Language Models with Influence Estimation
AI总结：
针对深度学习模型行为黑箱问题，提出结合外部知识库与大语言模型生成数据文本描述的 pipeline，引入影响估计筛选有效描述，通过跨模态迁移分类任务验证，零样本实验提升模型性能，助力理解数据与模型决策。
摘要：
深度学习模型在诸多领域取得成功，但其行为理解仍处于黑箱状态。多数现有可解释人工智能（XAI）方法聚焦于解释模型如何做出预测。相比之下，我们希望通过深度学习模型训练理解数据，并提出一种新颖方法，借助最常见的媒介——语言——来理解数据，以便人类轻松掌握。我们的方法提出一个利用外部知识库，通过大语言模型生成可解释数据的文本描述的 pipeline。然而，生成的数据描述可能包含无关信息，因此我们引入影响估计来选择信息最丰富的文本描述，并结合 CLIP 分数。此外，基于跨模态可迁移性现象，我们提出名为跨模态迁移分类的新颖基准任务，以检验文本描述的有效性。在零样本设置实验中，我们表明文本描述比其他基线描述更有效，且成功提升了仅在图像上训练的模型在所有九个图像分类数据集上的性能。这些结果得到 GPT-4o 评估的进一步支持。通过该方法，我们可深入了解模型决策过程的内在可解释性。
英文摘要：
Deep learning models have been successful in many areas but understanding their behaviors still remains a black-box. Most prior explainable AI (XAI) approaches have focused on interpreting and explaining how models make predictions. In contrast, we would like to understand how data can be explained with deep learning model training and propose a novel approach to understand the data via one of the most common media - language - so that humans can easily understand. Our approach proposes a pipeline to generate textual descriptions that can explain the data with large language models by incorporating external knowledge bases. However, generated data descriptions may still include irrelevant information, so we introduce to exploit influence estimation to choose the most informative textual descriptions, along with the CLIP score. Furthermore, based on the phenomenon of cross-modal transferability, we propose a novel benchmark task named cross-modal transfer classification to examine the effectiveness of our textual descriptions. In the experiment of zero-shot setting, we show that our textual descriptions are more effective than other baseline descriptions, and furthermore, we successfully boost the performance of the model trained only on images across all nine image classification datasets. These results are further supported by evaluation using GPT-4o. Through our approach, we may gain insights into the inherent interpretability of the decision-making process of the model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
