arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2007.12446cs.LGstat.ML
收藏

转移的离散性:量化表示之间的差异

Transferred Discrepancy: Quantifying the Difference Between Representations

Peking University(北京大学);Microsoft Research(微软研究院)

Yunzhen Feng, Runtian Zhai, Di He, Liwei Wang, Bin Dong

摘要:

了解神经网络捕获了什么信息是深度学习中的一个基本问题,研究不同的模型是否捕获了相似的特征是实现这一目标的第一步。以前的工作试图定义度量的特征矩阵来衡量两个模型之间的差异。然而,不同的度量标准有时会导致相互矛盾的结论,并且对于哪种度量标准适用于实践并没有达成共识。在这项工作中,我们提出了一种新的度量,超越了以前的方法。回想一下,使用学习到的表示的最实际的场景之一是将它们应用于下游任务。我们认为,我们应该根据类似的原则来设计度量。为此,我们引入了转移差异(TD),这是一个新的度量标准,它根据两种表示的下游任务性能定义了它们之间的差异。通过渐近分析,我们展示了TD如何与下游任务相关,以及以这种依赖于任务的方式定义度量的必要性。特别是,我们还表明,在特定条件下,TD度量是密切相关的以前的度量。我们的实验表明,TD可以为不同的下游任务提供细粒度的信息,并且对于从不同初始化训练的模型,学习的特征在下游任务预测方面是不一样的。我们发现,TD也可以用来评估不同的培训策略的有效性。例如,我们证明了用适当的数据增强训练的模型可以提高泛化能力,在TD方面捕获更多相似的特征,而那些用数据增强训练的模型则不会损害泛化能力。这表明,一种训练策略,导致更强大的代表性,也训练模型,更好地推广。

英文摘要:

Understanding what information neural networks capture is an essential problem in deep learning, and studying whether different models capture similar features is an initial step to achieve this goal. Previous works sought to define metrics over the feature matrices to measure the difference between two models. However, different metrics sometimes lead to contradictory conclusions, and there has been no consensus on which metric is suitable to use in practice. In this work, we propose a novel metric that goes beyond previous approaches. Recall that one of the most practical scenarios of using the learned representations is to apply them to downstream tasks. We argue that we should design the metric based on a similar principle. For that, we introduce the transferred discrepancy (TD), a new metric that defines the difference between two representations based on their downstream-task performance. Through an asymptotic analysis, we show how TD correlates with downstream tasks and the necessity to define metrics in such a task-dependent fashion. In particular, we also show that under specific conditions, the TD metric is closely related to previous metrics. Our experiments show that TD can provide fine-grained information for varied downstream tasks, and for the models trained from different initializations, the learned features are not the same in terms of downstream-task predictions. We find that TD may also be used to evaluate the effectiveness of different training strategies. For example, we demonstrate that the models trained with proper data augmentations that improve the generalization capture more similar features in terms of TD, while those with data augmentations that hurt the generalization will not. This suggests a training strategy that leads to more robust representation also trains models that generalize better.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑