arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2112.10071eess.IVcs.CV
收藏

一种新的人机两用图像编解码器

A New Image Codec Paradigm for Human and Machine Uses

Shandong Normal University(山东师范大学);Nanyang Technological University(南洋理工大学);Alibaba-NTU Singapore Joint Research Institute(阿里巴巴-南洋理工大学新加坡联合研究院);Alibaba Cloud Business Group(阿里云事业群)

Sien Chen, Jian Jin, Lili Meng, Weisi Lin, Zhuo Chen, Tsui-Shan Chang, Zhengguang Li, Huaxiang Zhang

摘要:

随着物联网(AIoT)的发展,大量的视觉数据,例如,图像和视频,都是我们日常工作和生活中产生的。这些视觉数据不仅用于人类观看或理解,而且用于机器分析或决策,例如,智能监控、自动驾驶汽车和许多其他智慧城市应用。为此,在这项工作中提出了一种新的图像编解码器的范式,为人类和机器使用。首先,利用神经网络提取高层实例分割图和低层信号特征。然后,实例分割图被进一步表示为具有所提出的16位灰度表示的轮廓。之后,16位灰度轮廓和信号特征都用无损编解码器编码。同时,设计并训练了一个图像预测器,实现了具有16位灰度轮廓和信号特征的一般质量的图像重建。最后,对预测图像和原始图像之间的残差映射进行有损压缩,用于高质量的图像重建。通过这样的设计,一方面,我们可以实现可伸缩的图像压缩,以满足不同人类消费的要求;另一方面,我们可以直接在解码器侧用解码的16位灰度轮廓实现几个机器视觉任务,例如,目标分类、检测和分割。实验结果表明,所提出的编解码器实现了与大多数基于学习的编解码器相当的结果,并且优于传统的编解码器(例如,BPG和JPEG 2000)在PSNR和MS-SSIM方面进行图像重建。同时,它在目标检测和分割的mAP方面优于现有的编解码器。

英文摘要:

With the AI of Things (AIoT) development, a huge amount of visual data, e.g., images and videos, are produced in our daily work and life. These visual data are not only used for human viewing or understanding but also for machine analysis or decision-making, e.g., intelligent surveillance, automated vehicles, and many other smart city applications. To this end, a new image codec paradigm for both human and machine uses is proposed in this work. Firstly, the high-level instance segmentation map and the low-level signal features are extracted with neural networks. Then, the instance segmentation map is further represented as a profile with the proposed 16-bit gray-scale representation. After that, both 16-bit gray-scale profile and signal features are encoded with a lossless codec. Meanwhile, an image predictor is designed and trained to achieve the general-quality image reconstruction with the 16-bit gray-scale profile and signal features. Finally, the residual map between the original image and the predicted one is compressed with a lossy codec, used for high-quality image reconstruction. With such designs, on the one hand, we can achieve scalable image compression to meet the requirements of different human consumption; on the other hand, we can directly achieve several machine vision tasks at the decoder side with the decoded 16-bit gray-scale profile, e.g., object classification, detection, and segmentation. Experimental results show that the proposed codec achieves comparable results as most learning-based codecs and outperforms the traditional codecs (e.g., BPG and JPEG2000) in terms of PSNR and MS-SSIM for image reconstruction. At the same time, it outperforms the existing codecs in terms of the mAP for object detection and segmentation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑