arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.15280cs.LGstat.ML
收藏

神经归一化压缩距离与压缩和分类之间的脱节

Neural Normalized Compression Distance and the Disconnect Between Compression and Classification

University of Maryland, Baltimore County(马里兰大学巴尔的摩县分校);Booz Allen Hamilton(博兹艾伦汉密尔顿公司)

John Hurwitz, Charles Nicholas, Edward Raff

AI总结:

本文通过将大语言模型用作无损压缩器构建神经归一化压缩距离,发现分类准确率不能仅由压缩率预测,揭示了压缩与分类之间的理解脱节。

摘要:

人们通常普遍理解,预测性分类与压缩在信息论中是内在相关的概念。事实上,许多深度学习方法被解释为在学习某种压缩,并且更好的压缩会带来更好的性能。我们通过归一化压缩距离(NCD)来审视这一假设,NCD 明确依赖压缩作为测量序列之间相似性的手段,从而实现最近邻分类。通过将流行的大语言模型(LLMs)转变为无损压缩器,我们开发了一种神经 NCD,并将 LLMs 与诸如 gzip 之类的经典通用算法进行比较。在此过程中,我们发现分类准确率不能仅凭压缩率来预测,同时还存在当前理解未能预测的其他经验性异常现象。我们的结果表明,我们关于神经网络“压缩”意味着什么以及有效分类需要什么条件的直觉,尚未得到充分理解。

英文摘要:

It is generally well understood that predictive classification and compression are intrinsically related concepts in information theory. Indeed, many deep learning methods are explained as learning a kind of compression, and that better compression leads to better performance. We interrogate this hypothesis via the Normalized Compression Distance (NCD), which explicitly relies on compression as the means of measuring similarity between sequences and thus enables nearest-neighbor classification. By turning popular large language models (LLMs) into lossless compressors, we develop a Neural NCD and compare LLMs to classic general-purpose algorithms like gzip. In doing so, we find that classification accuracy is not predictable by compression rate alone, among other empirical aberrations not predicted by current understanding. Our results imply that our intuition on what it means for a neural network to ``compress'' and what is needed for effective classification are not yet well understood.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑