增量序列标注:两种偏移的故事
Incremental Sequence Labeling: A Tale of Two Shifts
AI总结:
本文针对增量序列标注中的两种语义偏移(E2O 和 O2E),提出 IS3 框架,通过知识蒸馏和去偏损失缓解灾难性遗忘,在三个数据集上显著优于现有方法。
摘要:
增量序列标注任务涉及随时间不断学习新类别,同时保留对旧类别的知识。我们的研究识别出两种显著的语义偏移:E2O(模型将旧实体误标为非实体)和 O2E(模型将非实体或旧实体标为新实体)。以往研究主要聚焦于解决 E2O 问题,忽视了 O2E 问题。这种忽视导致模型在学习过程中偏向于将新数据样本分类为新类别。为应对这些挑战,我们提出了一种新颖的框架——无语义偏移的增量序列标注(IS3)。受所识别的语义偏移(E2O 和 O2E)启发,IS3 旨在缓解模型中的灾难性遗忘。针对 E2O 问题,我们使用知识蒸馏来保持模型对旧实体的判别能力。同时,为解决 O2E 问题,我们通过去偏损失和优化层面来减轻模型对新实体的偏向。我们在三个数据集上、多种增量设置下进行的实验评估表明,IS3 的性能显著优于先前最先进的方法。数据、代码和脚本可在 https://github.com/zzz47zzz/codebase-for-incremental-learning-with-llm 公开获取。
英文摘要:
The incremental sequence labeling task involves continuously learning new classes over time while retaining knowledge of the previous ones. Our investigation identifies two significant semantic shifts: E2O (where the model mislabels an old entity as a non-entity) and O2E (where the model labels a non-entity or old entity as a new entity). Previous research has predominantly focused on addressing the E2O problem, neglecting the O2E issue. This negligence results in a model bias towards classifying new data samples as belonging to the new class during the learning process. To address these challenges, we propose a novel framework, Incremental Sequential Labeling without Semantic Shifts (IS3). Motivated by the identified semantic shifts (E2O and O2E), IS3 aims to mitigate catastrophic forgetting in models. As for the E2O problem, we use knowledge distillation to maintain the model's discriminative ability for old entities. Simultaneously, to tackle the O2E problem, we alleviate the model's bias towards new entities through debiased loss and optimization levels. Our experimental evaluation, conducted on three datasets with various incremental settings, demonstrates the superior performance of IS3 compared to the previous state-of-the-art method by a significant margin.The data, code, and scripts are publicly available at https://github.com/zzz47zzz/codebase-for-incremental-learning-with-llm.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。