A New Image Codec Paradigm for Human and Machine Uses
Shandong Normal University(山东师范大学);Nanyang Technological University(南洋理工大学);Alibaba-NTU Singapore Joint Research Institute(阿里巴巴-南洋理工大学新加坡联合研究院);Alibaba Cloud Business Group(阿里云事业群)
With the AI of Things (AIoT) development, a huge amount of visual data, e.g., images and videos, are produced in our daily work and life. These visual data are not only used for human viewing or understanding but also for machine analysis or decision-making, e.g., intelligent surveillance, automated vehicles, and many other smart city applications. To this end, a new image codec paradigm for both human and machine uses is proposed in this work. Firstly, the high-level instance segmentation map and the low-level signal features are extracted with neural networks. Then, the instance segmentation map is further represented as a profile with the proposed 16-bit gray-scale representation. After that, both 16-bit gray-scale profile and signal features are encoded with a lossless codec. Meanwhile, an image predictor is designed and trained to achieve the general-quality image reconstruction with the 16-bit gray-scale profile and signal features. Finally, the residual map between the original image and the predicted one is compressed with a lossy codec, used for high-quality image reconstruction. With such designs, on the one hand, we can achieve scalable image compression to meet the requirements of different human consumption; on the other hand, we can directly achieve several machine vision tasks at the decoder side with the decoded 16-bit gray-scale profile, e.g., object classification, detection, and segmentation. Experimental results show that the proposed codec achieves comparable results as most learning-based codecs and outperforms the traditional codecs (e.g., BPG and JPEG2000) in terms of PSNR and MS-SSIM for image reconstruction. At the same time, it outperforms the existing codecs in terms of the mAP for object detection and segmentation.
Tell Codec What Worth Compressing: Semantically Disentangled Image Coding for Machine with LMMs
Shanghai Jiao Tong University(上海交通大学);Ningbo Institute of Digital Twin(宁波数字孪生研究院);Eastern Institute of Technology(东方理工大学);University of Science and Technology of China(中国科学技术大学);Yokohama National University(横滨国立大学)
Institute for AI Industry Research, Tsinghua University(清华大学人工智能产业研究院);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);SenseTime Research(商汤科技研究院);The Chinese University of Hong Kong(香港中文大学);School of Vehicle and Mobility, Tsinghua University(清华大学车辆与 mobility学院)