arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1709.02279cs.CL
收藏

语言模型训练数据的选择

Cynical Selection of Language Model Training Data

Amittai Axelrod

摘要:

Moore-Lewis的“智能选择语言模型训练数据“方法是一种非常有效、廉价、高效.也存在结构性问题。(1)该方法通过将在域内语料库和域外语料库(或数据池)上训练的语言模型相互播放来定义相关性。这一强有力的想法--也是我们要保留的--将两个语料库视为一个光谱的对立两端。这种缺乏细微差别的情况不允许两个语料库非常相似。在极端的情况下,来自相同分布的所有句子的Moore-Lewis得分都为零,因此没有结果排名。(2)选择的句子不能保证能够对域内数据进行建模,甚至也不能覆盖域内数据。它们只是受到领域模型的欢迎;这是必要的,但还不够。(3)没有办法知道选择的最佳句子数量是多少,除非选择各种阈值并构建系统。我们提出了一个贪婪的,懒惰的,近似的,一般有效的信息理论的方法来实现相同的目标,只使用词汇量。该方法具有以下特点:(1)对两个语料库的差异程度具有响应性。(2)快速达到接近最佳的词汇覆盖率。(3)考虑已经选择的内容。(4)不涉及定义任何类型的域,也不涉及定义任何类型的分类器。(6)他知道什么时候停下来。这种方法可以作为一种内在意义的相似性度量,因为它测量了通过将一个文本添加到另一个文本中所获得的信息位。

英文摘要:

The Moore-Lewis method of "intelligent selection of language model training data" is very effective, cheap, efficient... and also has structural problems. (1) The method defines relevance by playing language models trained on the in-domain and the out-of-domain (or data pool) corpora against each other. This powerful idea-- which we set out to preserve-- treats the two corpora as the opposing ends of a single spectrum. This lack of nuance does not allow for the two corpora to be very similar. In the extreme case where the come from the same distribution, all of the sentences have a Moore-Lewis score of zero, so there is no resulting ranking. (2) The selected sentences are not guaranteed to be able to model the in-domain data, nor to even cover the in-domain data. They are simply well-liked by the in-domain model; this is necessary, but not sufficient. (3) There is no way to tell what is the optimal number of sentences to select, short of picking various thresholds and building the systems. We present a greedy, lazy, approximate, and generally efficient information-theoretic method of accomplishing the same goal using only vocabulary counts. The method has the following properties: (1) Is responsive to the extent to which two corpora differ. (2) Quickly reaches near-optimal vocabulary coverage. (3) Takes into account what has already been selected. (4) Does not involve defining any kind of domain, nor any kind of classifier. (6) Knows approximately when to stop. This method can be used as an inherently-meaningful measure of similarity, as it measures the bits of information to be gained by adding one text to another.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑