arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2211.10086cs.CLcs.DL
收藏

元数据可能使语言模型更好

Metadata Might Make Language Models Better

The Alan Turing Institute(艾伦·图灵研究所);British Library(大英图书馆)

Kaspar Beelen, Daniel van Strien

摘要:

本文讨论了在历史集合上训练语言模型时包含元数据的好处。使用19世纪的报纸作为案例研究,我们扩展了Rosin等人提出的时间掩蔽方法,2022,并比较不同的策略插入到一个掩蔽的语言模型的时间,政治和地理信息。在对增强的输入数据微调了几个DistilBERT之后,我们在一组评估任务上对这些模型进行了系统的评估:伪困惑,元数据掩码填充和监督分类。我们发现,显示相关的元数据的语言模型有一个有益的影响,甚至可能产生更强大和更公平的模型。

英文摘要:

This paper discusses the benefits of including metadata when training language models on historical collections. Using 19th-century newspapers as a case study, we extend the time-masking approach proposed by Rosin et al., 2022 and compare different strategies for inserting temporal, political and geographical information into a Masked Language Model. After fine-tuning several DistilBERT on enhanced input data, we provide a systematic evaluation of these models on a set of evaluation tasks: pseudo-perplexity, metadata mask-filling and supervised classification. We find that showing relevant metadata to a language model has a beneficial impact and may even produce more robust and fairer models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑