发表机构
Tennessee State University; Enosis Solutions; North Carolina Agricultural and Technical University（田纳西州立大学; 伊诺西斯解决方案公司; 北卡罗来纳农工大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究针对表格数据学习，提出注意力头重要性评分方法，经40个数据集实验验证，可优化Transformer架构效率，相关代码已公开。
AI 中文摘要
计算量大且不透明的深度学习模型可通过分析其数据转换过程得到更好的理解与优化。尽管深度Transformer已在计算机视觉和自然语言处理领域得到广泛研究，但其在表格数据中的应用仍相对未被充分探索。本文提出首个将重要性评分指标用于解释多头部Transformer模型学习表格数据的研究之一。在40个不同表格数据集上开展的实验表明，基于所提出的注意力头重要性评分，模型对头部删除具有鲁棒性。在72.5%的实验案例中，当逐步移除重要性评分最低的头部时，模型对性能下降的抵抗性最强；相反，若先移除最重要的注意力头，会导致分类性能出现最大幅度的下降。对6个注意力层的单个注意力头重要性评分的进一步分析显示，重要头部分散在各层中，无一致的层特定趋势。与图像和语言领域不同，在具有不同模式和特征空间的表格数据集中，单个注意力头的重要性存在显著差异。所提出的重要性评分可提升Transformer架构的效率并减少冗余，我们公开了用于测量单个注意力头重要性的源代码。
英文摘要
Computationally demanding and opaque deep learning models can be better understood and optimized by analyzing how they transform data. While deep transformers have been widely studied in computer vision and natural language processing, their application in tabular data remains relatively underexplored. This paper presents one of the first applications of an importance-scoring metric to interpret multi-head transformer models in learning from tabular data. Experiments conducted on 40 diverse tabular datasets demonstrate robustness to head drops based on the proposed head importance score. In 72.5\% of experimental examples, the model remains most resilient to performance drops when heads with the lowest importance scores are gradually removed. In contrast, removing the most important attention head first results in the greatest reduction in classification performance. A closer look at individual head importance scores across six attention layers reveals that important heads are scattered across layers, with no consistent layer-specific trends. In contrast to the image and language domains, the importance of individual attention heads varies considerably across tabular datasets with different schemas and feature spaces. The proposed importance score can improve efficiency and redundancy within transformer architectures. We make the source code for measuring the importance of individual attention heads publicly available.
