最小描述长度的信息论探测
Information-Theoretic Probing with Minimum Description Length
摘要：
为了衡量预训练的表示对某些语言属性的编码程度，通常使用探测器的准确性，即训练分类器以从表示中预测属性。尽管探针被广泛采用，但其准确性的差异未能充分反映表征的差异。例如，它们基本上不支持预先训练的表示而不是随机初始化的表示。类似地，当探测真正的语言标签和探测随机合成任务时，它们的准确性可以是相似的。为了看到这些随机基线在精度上的合理差异，以前的工作必须约束探针训练数据的量或其模型大小。相反，我们提出了一种替代标准的探测，信息理论探测与最小描述长度（MDL）。通过MDL探测，训练探测器预测标签被重新设计为教导它有效地传输数据。因此，感兴趣的测量从探针准确度改变为给定表示的标签的描述长度。除了探测质量之外，描述长度还评估实现质量所需的“工作量”。这种工作量的特征是（i）探测模型的大小，或者（ii）实现高质量所需的数据量。我们考虑两种方法来估计MDL，可以很容易地实现在标准的探测管道：变分编码和在线编码。我们表明，这些方法的结果一致，比标准探针更丰富，更稳定。
英文摘要：
To measure how well pretrained representations encode some linguistic property, it is common to use accuracy of a probe, i.e. a classifier trained to predict the property from the representations. Despite widespread adoption of probes, differences in their accuracy fail to adequately reflect differences in representations. For example, they do not substantially favour pretrained representations over randomly initialized ones. Analogously, their accuracy can be similar when probing for genuine linguistic labels and probing for random synthetic tasks. To see reasonable differences in accuracy with respect to these random baselines, previous work had to constrain either the amount of probe training data or its model size. Instead, we propose an alternative to the standard probes, information-theoretic probing with minimum description length (MDL). With MDL probing, training a probe to predict labels is recast as teaching it to effectively transmit the data. Therefore, the measure of interest changes from probe accuracy to the description length of labels given representations. In addition to probe quality, the description length evaluates "the amount of effort" needed to achieve the quality. This amount of effort characterizes either (i) size of a probing model, or (ii) the amount of data needed to achieve the high quality. We consider two methods for estimating MDL which can be easily implemented on top of the standard probing pipelines: variational coding and online coding. We show that these methods agree in results and are more informative and stable than the standard probes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
