arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.07225cs.AI
收藏

Text Guide:基于特征重要性的文本选择方法提高长文本分类质量

Text Guide: Improving the quality of long text classification by a text selection method based on feature importance

University of Central Florida(中佛罗里达大学);Massachusetts Institute of Technology(麻省理工学院);University of Warsaw(华沙大学);Taif University(塔伊夫大学);University of Louisville(路易斯维尔大学)

Krzysztof Fiok, Waldemar Karwowski, Edgar Gutierrez, Mohammad Reza Davahli, Maciej Wilamowski, Tareq Ahram, Awad Al-Juaid, Jozef Zurada

摘要:

在过去的十年中,文本分类方法的性能在小于512个标记的文本实例中有了很大的提高。由于分析较长文本实例的高计算成本,大多数研究现状的Transformer模型都采用了此限制。为了缓解这个问题,并改善较长文本的分类,研究人员试图解决计算成本的根本原因,并提出了对注意力机制的优化,这是每个Transformer模型的关键要素。在我们的研究中,我们不追求长文本分类的最终目标,即,一次分析整个文本实例的能力,同时以合理的计算成本保持高性能。相反,我们提出了一种名为“文本指南”的文本截断方法,其中原始文本长度被减少到预定义的限制,其方式可以提高朴素和半朴素方法的性能,同时保持较低的计算成本。文本指南受益于特征重要性的概念,这是一个来自可解释人工智能领域的概念。我们证明,文本指南可以用来提高最近专门为长文本分类设计的语言模型,如Longformer的性能。此外,我们发现参数优化是文本指南性能的关键,必须在方法部署之前进行。未来的实验可能会揭示这种新方法提供的额外好处。

英文摘要:

The performance of text classification methods has improved greatly over the last decade for text instances of less than 512 tokens. This limit has been adopted by most state-of-the-research transformer models due to the high computational cost of analyzing longer text instances. To mitigate this problem and to improve classification for longer texts, researchers have sought to resolve the underlying causes of the computational cost and have proposed optimizations for the attention mechanism, which is the key element of every transformer model. In our study, we are not pursuing the ultimate goal of long text classification, i.e., the ability to analyze entire text instances at one time while preserving high performance at a reasonable computational cost. Instead, we propose a text truncation method called Text Guide, in which the original text length is reduced to a predefined limit in a manner that improves performance over naive and semi-naive approaches while preserving low computational costs. Text Guide benefits from the concept of feature importance, a notion from the explainable artificial intelligence domain. We demonstrate that Text Guide can be used to improve the performance of recent language models specifically designed for long text classification, such as Longformer. Moreover, we discovered that parameter optimization is the key to Text Guide performance and must be conducted before the method is deployed. Future experiments may reveal additional benefits provided by this new method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑