Text Guide: Improving the quality of long text classification by a text selection method based on feature importance
University of Central Florida(中佛罗里达大学);Massachusetts Institute of Technology(麻省理工学院);University of Warsaw(华沙大学);Taif University(塔伊夫大学);University of Louisville(路易斯维尔大学)
Krzysztof Fiok, Waldemar Karwowski, Edgar Gutierrez, Mohammad Reza Davahli, Maciej Wilamowski, Tareq Ahram, Awad Al-Juaid, Jozef Zurada
备注:
This is the reviewed and accepted for publication version of the article by the IEEE Access Journal. One of the important modifications is publication of the code along with the paper. The code can be used to apply Text Guide to a data set of ones choice. The code is available at: https://github.com/krzysztoffiok/TextGuide
The performance of text classification methods has improved greatly over the last decade for text instances of less than 512 tokens. This limit has been adopted by most state-of-the-research transformer models due to the high computational cost of analyzing longer text instances. To mitigate this problem and to improve classification for longer texts, researchers have sought to resolve the underlying causes of the computational cost and have proposed optimizations for the attention mechanism, which is the key element of every transformer model. In our study, we are not pursuing the ultimate goal of long text classification, i.e., the ability to analyze entire text instances at one time while preserving high performance at a reasonable computational cost. Instead, we propose a text truncation method called Text Guide, in which the original text length is reduced to a predefined limit in a manner that improves performance over naive and semi-naive approaches while preserving low computational costs. Text Guide benefits from the concept of feature importance, a notion from the explainable artificial intelligence domain. We demonstrate that Text Guide can be used to improve the performance of recent language models specifically designed for long text classification, such as Longformer. Moreover, we discovered that parameter optimization is the key to Text Guide performance and must be conducted before the method is deployed. Future experiments may reveal additional benefits provided by this new method.
MESH-Harness: Self-Improving Agent Harnesses via Bandit-Guided Compositional Evolution
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));The Chinese University of Hong Kong(香港中文大学);Nanyang Technological University(南洋理工大学);Fudan University(复旦大学);National University of Singapore(新加坡国立大学);DeepWisdom(深度智慧)
REPEAT: Improving Uncertainty Estimation in Representation Learning Explainability
UiT The Arctic University of Norway(挪威北极大学);Technical University of Denmark(丹麦技术大学);Norwegian Computing Center(挪威计算中心);University of Copenhagen(哥本哈根大学)
Ten Words Only Still Help: Improving Black-Box AI-Generated Text Detection via Proxy-Guided Efficient Re-Sampling
Key Lab of Intelligent Information Processing of Chinese Academy of Sciences(中国科学院智能信息处理重点实验室);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Xi’an Jiaotong University(西安交通大学)
TinyDrop: Tiny Model Guided Token Dropping for Vision Transformers
School of Electrical and Electronic Engineering, University College Dublin, Ireland(电气与电子工程学院,都柏林大学学院,爱尔兰);College of Electronics and Information Engineering, Shenzhen University, China(电子与信息工程学院,深圳大学,中国)
Improving Low-Resource Knowledge Tracing Tasks by Supervised Pre-training and Importance Mechanism Fine-tuning
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Guangdong Institute of Smart Education, Jinan University(暨南大学广东智慧教育研究院);TAL Education Group(好未来教育集团)