Although attention mechanisms have become fundamental components of deep learning models, they are vulnerable to perturbations, which may degrade the prediction performance and model interpretability. Adversarial training (AT) for attention mechanisms has successfully reduced such drawbacks by considering adversarial perturbations. However, this technique requires label information, and thus, its use is limited to supervised settings. In this study, we explore the concept of incorporating virtual AT (VAT) into the attention mechanisms, by which adversarial perturbations can be computed even from unlabeled data. To realize this approach, we propose two general training techniques, namely VAT for attention mechanisms (Attention VAT) and "interpretable" VAT for attention mechanisms (Attention iVAT), which extend AT for attention mechanisms to a semi-supervised setting. In particular, Attention iVAT focuses on the differences in attention; thus, it can efficiently learn clearer attention and improve model interpretability, even with unlabeled data. Empirical experiments based on six public datasets revealed that our techniques provide better prediction performance than conventional AT-based as well as VAT-based techniques, and stronger agreement with evidence that is provided by humans in detecting important words in sentences. Moreover, our proposal offers these advantages without needing to add the careful selection of unlabeled data. That is, even if the model using our VAT-based technique is trained on unlabeled data from a source other than the target task, both the prediction performance and model interpretability can be improved.
句子理解的主动学习试图通过识别信息量最大的示例来降低注释成本。主动学习的常用方法在基于池的场景中使用不确定性或多样性采样。在这项工作中,为了结合预测不确定性和样本多样性,我们提出了主动学习的虚拟对抗扰动(VAPAL),这是一种不确定性-多样性组合框架,使用虚拟对抗扰动(Miyato et al.,2019)作为模型不确定性表示。VAPAL在四个句子理解数据集上的表现一直与强基线一样好,甚至更好:AGNEWS,IMDB,PUBMED和SST-2,为句子理解任务的主动学习提供了一个潜在的选择。
TAVAT: Token-Aware Virtual Adversarial Training for Language Understanding
Shanghai Key Laboratory of Intelligent Information Processing, Fudan University(复旦大学上海市智能信息处理重点实验室);School of Computer Science, Fudan University(复旦大学计算机科学技术学院)
Interpretable Adversarial Perturbation in Input Embedding Space for Text
Preferred Networks, Inc.(优选网络公司);Nara Institute of Science and Technology(奈良先端科学技术大学院大学);RIKEN AIP(理化学研究所先进智能项目中心);Tohoku University(东北大学);NTT Communication Science Laboratories(日本电信电话通信科学实验室)