自然语言处理中新颖性检测与适应的统一评估框架及其在作者归属中的实例化
A Unified Evaluation Framework for Novelty Detection and Accommodation in NLP with an Instantiation in Authorship Attribution
AI总结：
本文提出统一评估框架NoveltyTask，用于自然语言处理中的新颖性检测与适应，并在作者归属任务上实例化，实验显示现有方法性能较低，留有改进空间。
摘要：
最先进的自然语言处理模型已被证明在“封闭世界”设置中表现出卓越的性能，在这种设置中，评估集中的所有标签在训练时都是已知的。然而，在现实世界的设置中，经常观察到不属于任何已知类别的“新颖”实例。这使得处理新颖性的能力变得至关重要。为了在这一重要的“处理新颖性”领域开展系统性研究，我们引入了“NoveltyTask”，这是一个多阶段任务，用于评估系统在流水线式新颖性“检测”和“适应”任务上的性能。我们提供了NoveltyTask的数学公式，并将其实例化为作者归属任务，该任务涉及识别给定文本的正确作者。我们使用亚马逊评论语料库，为NoveltyTask编译了一个大型数据集（包含200位作者/标签的25万条实例）。我们进行了全面的实验，并探索了几种基线方法。我们的结果表明，这些方法取得了相当低的性能，使得任务具有挑战性，并留下了足够的改进空间。最后，我们相信我们的工作将鼓励在这一未被充分探索的处理新颖性领域进行研究，这是开发健壮系统的重要一步。
英文摘要：
State-of-the-art natural language processing models have been shown to achieve remarkable performance in 'closed-world' settings where all the labels in the evaluation set are known at training time. However, in real-world settings, 'novel' instances that do not belong to any known class are often observed. This renders the ability to deal with novelties crucial. To initiate a systematic research in this important area of 'dealing with novelties', we introduce 'NoveltyTask', a multi-stage task to evaluate a system's performance on pipelined novelty 'detection' and 'accommodation' tasks. We provide mathematical formulation of NoveltyTask and instantiate it with the authorship attribution task that pertains to identifying the correct author of a given text. We use Amazon reviews corpus and compile a large dataset (consisting of 250k instances across 200 authors/labels) for NoveltyTask. We conduct comprehensive experiments and explore several baseline methods for the task. Our results show that the methods achieve considerably low performance making the task challenging and leaving sufficient room for improvement. Finally, we believe our work will encourage research in this underexplored area of dealing with novelties, an important step en route to developing robust systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
