通过查找子网进行低复杂度探测
Low-Complexity Probing via Finding Subnetworks
摘要：
探测神经网络语言特性的主要方法是在模型的内部表示之上训练一个新的浅层多层感知器（MLP）。这种方法可以检测模型中编码的属性，但代价是添加了可能直接学习任务的新参数。相反，我们提出了一个减法修剪为基础的探针，在那里我们找到一个现有的子网络，执行感兴趣的语言任务。与MLP相比，子网络探测器在预训练模型上实现了更高的准确性，而在随机模型上实现了更低的准确性，因此它在寻找感兴趣的属性方面更好，但在自行学习方面较差。接下来，通过改变每个探针的复杂性，我们表明，子网探测帕累托主导的MLP探测，它实现了更高的准确性，任何预算的探测复杂性。最后，我们分析了不同任务之间的子网络，以定位每个任务的编码位置，我们发现较低级别的任务被捕获在较低的层中，重现了过去工作中的类似发现。
英文摘要：
The dominant approach in probing neural networks for linguistic properties is to train a new shallow multi-layer perceptron (MLP) on top of the model's internal representations. This approach can detect properties encoded in the model, but at the cost of adding new parameters that may learn the task directly. We instead propose a subtractive pruning-based probe, where we find an existing subnetwork that performs the linguistic task of interest. Compared to an MLP, the subnetwork probe achieves both higher accuracy on pre-trained models and lower accuracy on random models, so it is both better at finding properties of interest and worse at learning on its own. Next, by varying the complexity of each probe, we show that subnetwork probing Pareto-dominates MLP probing in that it achieves higher accuracy given any budget of probe complexity. Finally, we analyze the resulting subnetworks across various tasks to locate where each task is encoded, and we find that lower-level tasks are captured in lower layers, reproducing similar findings in past work.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
