arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.02790cs.CL
收藏

使用预训练语言模型的低资源摘要生成

Low Resource Summarization using Pre-trained Language Models

MCS(军事学院(注:MCS常见对应Military College of Signals,即信号军事学院,此处按原文缩写保留,直译军事学院))

Mubashir Munaf, Hammad Afzal, Naima Iltaf, Khawir Mahmood

AI总结:

该研究将 mBERT、mT5 等预训练模型适配于乌尔都语低资源摘要,构建 76.5k 新闻文章-摘要数据集,并提出体积更小、性能可比英语 SOTA 模型的 urT5。

摘要:

随着基于深度学习的人工神经网络模型的出现,自然语言处理(NLP)在文本数据处理的效率和准确性方面取得了显著提升。然而,相关研究大多局限于英语等高资源语言,低资源语言仍面临训练数据集以及模型资源匮乏的问题,甚至缺乏基线评估结果。考虑到低资源语言资源的有限可得性,我们提出了一种将基于自注意力 transformer 架构的模型(mBERT、mT5)适配用于低资源摘要生成的方法,并辅以在低资源语言乌尔都语中构建的新基线数据集(76.5k 个文章-摘要对)。选择新闻(一种公开可用的来源)作为应用领域,使所提出的方法有望复用于其他资源有限的语言。我们适配后的摘要模型 urT5 与 mT5 相比体积最多缩小 44.78%,能够有效捕捉低资源语言的上下文信息,其评估分数(最高达 46.35 ROUGE-1、77 BERTScore)与高资源语言英语中的最先进模型(XSUM 数据集上的 PEGASUS:47.21,BART:45.14)相当。所提出的方法为抽取式和生成式摘要提供了一种基线途径,并在资源有限的设置下取得了具有竞争力的评估结果。

英文摘要:

With the advent of Deep Learning based Artificial Neural Networks models, Natural Language Processing (NLP) has witnessed significant improvements in textual data processing in terms of its efficiency and accuracy. However, the research is mostly restricted to high-resource languages such as English and low-resource languages still suffer from a lack of available resources in terms of training datasets as well as models with even baseline evaluation results. Considering the limited availability of resources for low-resource languages, we propose a methodology for adapting self-attentive transformer-based architecture models (mBERT, mT5) for low-resource summarization, supplemented by the construction of a new baseline dataset (76.5k article, summary pairs) in a low-resource language Urdu. Choosing news (a publicly available source) as the application domain has the potential to make the proposed methodology useful for reproducing in other languages with limited resources. Our adapted summarization model \textit{urT5} with up to 44.78\% reduction in size as compared to \textit{mT5} can capture contextual information of low resource language effectively with evaluation score (up to 46.35 ROUGE-1, 77 BERTScore) at par with state-of-the-art models in high resource language English \textit{(PEGASUS: 47.21, BART: 45.14 on XSUM Dataset)}. The proposed method provided a baseline approach towards extractive as well as abstractive summarization with competitive evaluation results in a limited resource setup.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑