arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.07286cs.CLcs.AI
收藏

小型语言模型成为高效长文本提取器

Small Language Model Makes an Effective Long Text Extractor

Zhipu AI(智谱AI)

Yelin Chen, Fanjin Zhang, Jie Tang

AI总结:

针对长文本中长实体跨度提取的研究空白,本文提出轻量级基于跨度的NER方法SeNER,通过双向箭头注意力、LogN-Scaling及BiSPA机制减少冗余计算,在三个数据集上实现SOTA精度且GPU内存友好。

摘要:

命名实体识别(NER)是自然语言处理(NLP)中的基础问题。然而,从扩展文本(如主页)中提取较长实体跨度(如奖项)的任务几乎未被探索。当前NER方法主要分为两类:基于跨度的方法和基于生成的方法。基于跨度的方法需要枚举所有可能的词对跨度,然后对每个跨度进行分类,导致大量冗余计算和过度GPU内存使用。相比之下,基于生成的方法涉及提示或微调大型语言模型(LLMs)以适应下游NER任务,但这些方法难以准确生成较长跨度,且有效微调通常会产生显著时间成本。为解决这些挑战,本文提出一种轻量级基于跨度的NER方法SeNER,该方法结合双向箭头注意力机制与[CLS] token上的LogN-Scaling以有效嵌入长文本,并包含新颖的双向滑动窗口加号形注意力(BiSPA)机制,可显著减少冗余候选词对跨度,同时建模词对跨度间的交互。大量实验表明,该方法在三个长NER数据集上实现了最先进的提取精度,且能以GPU内存友好的方式从长文本中提取实体。代码:https://github.com/THUDM/scholar-profiling/tree/main/sener

英文摘要:

Named Entity Recognition (NER) is a fundamental problem in natural language processing (NLP). However, the task of extracting longer entity spans (e.g., awards) from extended texts (e.g., homepages) is barely explored. Current NER methods predominantly fall into two categories: span-based methods and generation-based methods. Span-based methods require the enumeration of all possible token-pair spans, followed by classification on each span, resulting in substantial redundant computations and excessive GPU memory usage. In contrast, generation-based methods involve prompting or fine-tuning large language models (LLMs) to adapt to downstream NER tasks. However, these methods struggle with the accurate generation of longer spans and often incur significant time costs for effective fine-tuning. To address these challenges, this paper introduces a lightweight span-based NER method called SeNER, which incorporates a bidirectional arrow attention mechanism coupled with LogN-Scaling on the [CLS] token to embed long texts effectively, and comprises a novel bidirectional sliding-window plus-shaped attention (BiSPA) mechanism to reduce redundant candidate token-pair spans significantly and model interactions between token-pair spans simultaneously. Extensive experiments demonstrate that our method achieves state-of-the-art extraction accuracy on three long NER datasets and is capable of extracting entities from long texts in a GPU-memory-friendly manner. Code: https://github.com/THUDM/scholar-profiling/tree/main/sener

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑