arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2009.07368cs.LGcs.AIstat.ML
收藏

通过学习低损失预测器的复杂性来评估表示

Evaluating representations by the complexity of learning low-loss predictors

Courant Institute, New York University(纽约大学柯朗研究所);Columbia University(哥伦比亚大学)

William F. Whitney, Min Jae Song, David Brandfonbrener, Jaan Altosaar, Kyunghyun Cho

摘要:

我们考虑的问题,评估表示的数据用于解决下游任务。我们建议通过在感兴趣的任务上实现低损失的表示上学习预测器的复杂性来衡量表示的质量,并引入两种方法,剩余描述长度(SDL)和$\varepsilon$样本复杂性($\varepsilon$ SC)。与先前的方法相比,这些方法测量关于存在于特定量的数据中的最佳预测器的信息量,我们的方法测量从数据中恢复最佳预测器的近似值所需的信息量,直到指定的公差。我们提出了一个框架来比较这些方法的基础上绘制的验证损失与评估数据集的大小(“损失数据”曲线)。现有的措施,如互信息和最小描述长度探针,对应于沿损失数据曲线的数据轴的切片和积分,而我们的对应于沿损失轴的切片和积分。我们提供了真实数据的实验,以比较这些方法中的每一种在不同大小的数据集上的行为,以及在https://github.com/willwhitney/reprieve上用于表示评估的高性能开源库。

英文摘要:

We consider the problem of evaluating representations of data for use in solving a downstream task. We propose to measure the quality of a representation by the complexity of learning a predictor on top of the representation that achieves low loss on a task of interest, and introduce two methods, surplus description length (SDL) and $\varepsilon$ sample complexity ($\varepsilon$SC). In contrast to prior methods, which measure the amount of information about the optimal predictor that is present in a specific amount of data, our methods measure the amount of information needed from the data to recover an approximation of the optimal predictor up to a specified tolerance. We present a framework to compare these methods based on plotting the validation loss versus evaluation dataset size (the "loss-data" curve). Existing measures, such as mutual information and minimum description length probes, correspond to slices and integrals along the data axis of the loss-data curve, while ours correspond to slices and integrals along the loss axis. We provide experiments on real data to compare the behavior of each of these methods over datasets of varying size along with a high performance open source library for representation evaluation at https://github.com/willwhitney/reprieve.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑