arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.00727cs.CL
收藏

共指消解模型的受控再评估

A Controlled Reevaluation of Coreference Resolution Models

McGill University(麦吉尔大学);Mila - Quebec AI Institute(米拉 - 魁北克人工智能研究所)

Ian Porada, Xiyuan Zou, Jackie Chi Kit Cheung

AI总结:

通过控制预训练语言模型等设计决策,系统评估五个共指消解模型,发现基于编码器的模型在准确性和速度上优于解码器模型,且最古老模型跨领域泛化最佳,控制语言模型可消除大部分但非全部F1提升。

摘要:

所有最先进的共指消解(CR)模型都涉及对预训练语言模型的微调。由于缺乏标准化的实验设置,一个CR模型相对于另一个模型的优越性能究竟归因于语言模型的选择还是其他因素(如任务特定架构),往往难以甚至无法确定。为解决这一模糊性,我们系统评估了五个CR模型,并控制了某些设计决策,包括每个模型所使用的预训练语言模型。在控制语言模型规模时,基于编码器的CR模型在准确性和推理速度方面均优于较新的基于解码器的模型。令人惊讶的是,在基于编码器的CR模型中,较新的模型并非总是更准确,而我们测试的最古老的CR模型在跨领域文本类型上的泛化能力最佳。我们得出结论:控制语言模型的选择可以消除过去五年报告的大部分F1分数提升,但并非全部。

英文摘要:

All state-of-the-art coreference resolution (CR) models involve finetuning a pretrained language model. Whether the superior performance of one CR model over another is due to the choice of language model or other factors, such as the task-specific architecture, is difficult or impossible to determine due to lack of a standardized experimental setup. To resolve this ambiguity, we systematically evaluate five CR models and control for certain design decisions including the pretrained language model used by each. When controlling for language model size, encoder-based CR models outperform more recent decoder-based models in terms of both accuracy and inference speed. Surprisingly, among encoder-based CR models, more recent models are not always more accurate, and the oldest CR model that we test generalizes the best to out-of-domain textual genres. We conclude that controlling for the choice of language model reduces most, but not all, of the increase in F1 score reported in the past five years.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑