arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2108.13759cs.CL
收藏

享受显着性:用词显着性实现更好的基于转换器的忠实解释

Enjoy the Salience: Towards Better Transformer-based Faithful Explanations with Word Salience

University of Sheffield(谢菲尔德大学)

George Chrysostomou, Nikolaos Aletras

摘要:

经过预训练的基于transformer的模型,如BERT,在适应一系列自然语言处理任务时,已表现出最先进的预测性能。一个开放的问题是如何提高这些模型的预测解释(理由)的可信度。在本文中,我们假设从训练数据中提取的先验显著信息可以补充模型在对下游任务进行微调时学到的特定于任务的信息。通过这种方式,我们的目标是通过提出SaLoss来帮助BERT在进行预测时不要忘记为信息输入标记分配重要性; SaLoss是一种辅助损失函数,用于在训练期间引导多头注意力机制接近使用TextRank先验提取的显著信息。在五个数据集上进行的解释忠诚度实验表明,与香草BERT相比,使用SaLoss训练的模型在四种不同的特征归因方法中始终提供更忠实的解释。使用从香草BERT和SaLoss模型中提取的基本原理来训练固有忠实的分类器,我们进一步表明,后者在下游任务中具有更高的预测性能。

英文摘要:

Pretrained transformer-based models such as BERT have demonstrated state-of-the-art predictive performance when adapted into a range of natural language processing tasks. An open problem is how to improve the faithfulness of explanations (rationales) for the predictions of these models. In this paper, we hypothesize that salient information extracted a priori from the training data can complement the task-specific information learned by the model during fine-tuning on a downstream task. In this way, we aim to help BERT not to forget assigning importance to informative input tokens when making predictions by proposing SaLoss; an auxiliary loss function for guiding the multi-head attention mechanism during training to be close to salient information extracted a priori using TextRank. Experiments for explanation faithfulness across five datasets, show that models trained with SaLoss consistently provide more faithful explanations across four different feature attribution methods compared to vanilla BERT. Using the rationales extracted from vanilla BERT and SaLoss models to train inherently faithful classifiers, we further show that the latter result in higher predictive performance in downstream tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑