arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.07002cs.CLcs.AI
收藏

文本分类任务中的经典分布外检测方法基准

Classical Out-of-Distribution Detection Methods Benchmark in Text Classification Tasks

Wroclaw University of Science and Technology(弗罗茨瓦夫科技大学);Alphamoon Ltd.(Alphamoon有限公司);Tooploox Ltd.(Tooploox有限公司)

Mateusz Baran, Joanna Baran, Mateusz Wójcik, Maciej Zięba, Adam Gonczarek

AI总结:

本文评估八种无需额外数据或模型修改即可集成的经典 OOD 检测方法,构建可复现实验环境,揭示其在背景偏移和随机词序场景下的不足。

摘要:

最先进的模型在受控环境中可以表现良好,但在面对分布外(OOD)样本时往往表现不佳,这使得 OOD 检测成为 NLP 系统的关键组成部分。在本文中,我们重点强调 NLP 中现有 OOD 检测方法的局限性。具体而言,我们评估了八种可轻松集成到现有 NLP 系统中、且不需要额外 OOD 数据或模型修改的 OOD 检测方法。我们的贡献之一是提供一个结构良好的研究环境,使结果能够完全复现。此外,我们的分析表明,现有的 NLP 任务 OOD 检测方法尚不够敏感,无法捕获以各类分布偏移为特征的全部样本。特别具有挑战性的测试场景出现在背景偏移以及域内文本中单词顺序被随机打乱的情况下。这凸显出未来需要为 NLP 问题开发更有效的 OOD 检测方法,而我们的工作为该领域的进一步研究提供了定义明确的基础。

英文摘要:

State-of-the-art models can perform well in controlled environments, but they often struggle when presented with out-of-distribution (OOD) examples, making OOD detection a critical component of NLP systems. In this paper, we focus on highlighting the limitations of existing approaches to OOD detection in NLP. Specifically, we evaluated eight OOD detection methods that are easily integrable into existing NLP systems and require no additional OOD data or model modifications. One of our contributions is providing a well-structured research environment that allows for full reproducibility of the results. Additionally, our analysis shows that existing OOD detection methods for NLP tasks are not yet sufficiently sensitive to capture all samples characterized by various types of distributional shifts. Particularly challenging testing scenarios arise in cases of background shift and randomly shuffled word order within in domain texts. This highlights the need for future work to develop more effective OOD detection approaches for the NLP problems, and our work provides a well-defined foundation for further research in this area.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑