One of the most studied problems in machine learning is finding reasonable constraints that guarantee the generalization of a learning algorithm. These constraints are usually expressed as some simplicity assumptions on the target. For instance, in the Vapnik-Chervonenkis (VC) theory the space of possible hypotheses is considered to have a limited VC dimension. In this paper, the constraint on the entropy $H(X)$ of the input variable $X$ is studied as a simplicity assumption. It is proven that the sample complexity to achieve an $ε$-$δ$ Probably Approximately Correct (PAC) hypothesis is bounded by $\frac{2^{ \left.6H(X)\middle/ε\right.}+\log{\frac{1}δ}}{ε^2}$ which is sharp up to the $\frac{1}{ε^2}$ factor. Morever, it is shown that if a feature learning process is employed to learn the compressed representation from the dataset, this bound no longer exists. These findings have important implications on the Information Bottleneck (IB) theory which had been utilized to explain the generalization power of Deep Neural Networks (DNNs), but its applicability for this purpose is currently under debate by researchers. In particular, this is a rigorous proof for the previous heuristic that compressed representations are exponentially easier to be learned. However, our analysis pinpoints two factors preventing the IB, in its current form, to be applicable in studying neural networks. Firstly, the exponential dependence of sample complexity on $\frac{1}ε$, which can lead to a dramatic effect on the bounds in practical applications when $ε$ is small. Secondly, our analysis reveals that arguments based on input compression are inherently insufficient to explain generalization of methods like DNNs in which the features are also learned using available data.
DCT-CompCNN: A Novel Image Classification Network Using JPEG Compressed DCT Coefficients
Indian Institute of Information Technology, Allahabad(印度理工学院安拉阿巴德分校);Madan Mohan Malaviya University of Technology(马丹·莫汉·马拉维雅理工大学)
卷积神经网络(CNN)在图像处理和计算机视觉领域的普及促使全球的研究人员和工业专家以高精度解决不同的挑战。训练CNN分类器的最简单方法是直接将原始RGB像素图像输入网络。然而,如果我们打算直接使用其压缩数据对图像进行分类,则相同的方法可能无法更好地工作,例如JPEG压缩图像的情况。本研究探讨如何修改JPEG压缩数据的输入表示,然后输入到CNN中。该架构被称为DCT-CompCNN。这种新的方法表明,CNN也可以用JPEG压缩的DCT系数进行训练,并且与传统的CNN方法相比,可以产生更好的性能。在公共图像分类数据集(如Dog Vs Cat和CIFAR-10数据集)上,使用现有的ResNet-50架构和建议的DCT-CompCNN架构测试了修改后的输入表示的效率,报告了更好的性能
Nonconvex Nonsmooth Low-Rank Minimization for Generalized Image Compressed Sensing via Group Sparse Representation
College of Electronic and Optical Engineering(电子与光学工程学院);College of Microelectronics(微电子学院);Nanjing University of Posts and Telecommunications(南京邮电大学);College of Telecommunications & Information Engineering(通信与信息工程学院)