arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.14899cs.CL
收藏

基于检索的文本选择以解决分类中的类不平衡数据

Retrieval-based Text Selection for Addressing Class-Imbalanced Data in Classification

Virginia Tech(弗吉尼亚理工大学)

Sareh Ahmadi, Aditya Shah, Edward Fox

AI总结:

本文提出一种基于检索的文本选择方法,利用SHAP构建查询集,结合向量搜索、语义搜索和机器学习分类器,在标注数量受限时解决类不平衡问题,实验证明提高了少数类别的F1分数。

摘要:

本文针对在人力资源限制导致标注数量受限的情况下,使用检索方法在文本分类中选择一组待标注文本的问题。另一个解决的挑战是处理具有少量正实例的二分类类别,这反映了严重的类不平衡。在我们的情境中,标注过程跨越较长时间段,待标注文本的选择可以分批进行,前一批的标注结果指导下一批的选择。为解决这些挑战,本文提出利用SHAP来构建用于Elasticsearch和语义搜索的高质量查询集,以尝试识别有助于缓解类不平衡的最优待标注文本集。该方法在描述可能未来事件的提示文本集上进行了测试,这些文本由参与旨在帮助管理肥胖和糖尿病研究的参与者构建。我们引入了一种有效的方法,用于选择少量待标注文本并构建高质量的分类器。我们整合了向量搜索、语义搜索和机器学习分类器以产生良好的解决方案。我们的实验表明,在二分类中少数类别的F1分数得到了改进。

英文摘要:

This paper addresses the problem of selecting of a set of texts for annotation in text classification using retrieval methods when there are limits on the number of annotations due to constraints on human resources. An additional challenge addressed is dealing with binary categories that have a small number of positive instances, reflecting severe class imbalance. In our situation, where annotation occurs over a long time period, the selection of texts to be annotated can be made in batches, with previous annotations guiding the choice of the next set. To address these challenges, the paper proposes leveraging SHAP to construct a quality set of queries for Elasticsearch and semantic search, to try to identify optimal sets of texts for annotation that will help with class imbalance. The approach is tested on sets of cue texts describing possible future events, constructed by participants involved in studies aimed to help with the management of obesity and diabetes. We introduce an effective method for selecting a small set of texts for annotation and building high-quality classifiers. We integrate vector search, semantic search, and machine learning classifiers to yield a good solution. Our experiments demonstrate improved F1 scores for the minority classes in binary classification.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑