The proliferation of deep learning-based machine vision applications has given rise to a new type of compression, so called video coding for machine (VCM). VCM differs from traditional video coding in that it is optimized for machine vision performance instead of human visual quality. In the feature compression track of MPEG-VCM, multi-scale features extracted from images are subject to compression. Recent feature compression works have demonstrated that the versatile video coding (VVC) standard-based approach can achieve a BD-rate reduction of up to 96% against MPEG-VCM feature anchor. However, it is still sub-optimal as VVC was not designed for extracted features but for natural images. Moreover, the high encoding complexity of VVC makes it difficult to design a lightweight encoder without sacrificing performance. To address these challenges, we propose a novel multi-scale feature compression method that enables both the end-to-end optimization on the extracted features and the design of lightweight encoders. The proposed model combines a learnable compressor with a multi-scale feature fusion network so that the redundancy in the multi-scale features is effectively removed. Instead of simply cascading the fusion network and the compression network, we integrate the fusion and encoding processes in an interleaved way. Our model first encodes a larger-scale feature to obtain a latent representation and then fuses the latent with a smaller-scale feature. This process is successively performed until the smallest-scale feature is fused and then the encoded latent at the final stage is entropy-coded for transmission. The results show that our model outperforms previous approaches by at least 52% BD-rate reduction and has $\times5$ to $\times27$ times less encoding time for object detection...
Perceptual Video Coding for Machines via Satisfied Machine Ratio Modeling
National Engineering Research Center of Visual Technology, School of Computer Science, Peking University(北京大学计算机学院视觉技术国家工程研究中心);University of Chinese Academy of Sciences(中国科学院大学);Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Peng Cheng Laboratory(鹏城实验室)
Video Coding for Machine: Compact Visual Representation Compression for Intelligent Collaborative Analytics
Peking University(北京大学)
机器视频编码(Video Coding for Machines,VCM)致力于在一定程度上弥合视频/图像压缩和特征压缩这两条各自独立的研究路径,并试图从高精度机器视觉和全保真度人类视觉的统一角度出发,共同优化紧凑性和效率。在本文中,我们总结了VCM方法和哲学的基础上现有的学术界和工业界的努力。音圈调制的发展遵循一般的率失真优化,并建立了关键模块或技术的分类。从以前的作品中,它表明,虽然现有的作品试图揭示的性质,可扩展表示位处理机器和人类视觉任务时,仍然是一个罕见的研究在低比特率表示的一般性,因此如何支持各种视觉分析任务。因此,我们研究了一种新的视觉信息压缩的分析分类问题,以加强从多个任务中提取的紧凑的视觉表示的视觉分析的能力。一个新的视角与压缩的任务关系被重新审视。通过考虑不同机器视觉任务(例如高级语义和中级几何相关)之间的可转移性,我们的目标是以低比特率联合支持多个任务。特别地,为了缩小从像素提取的神经网络生成的特征与各种机器视觉特征/标签(例如,场景类、分割标签)之间的维度差距,码本超先验被设计为压缩神经网络生成的特征。实验结果表明,这种新的超先验模型能够更准确地估计信号熵,从而提高特征压缩效率,进一步研究不同任务间提取紧凑特征的粒度。
我们提出了一种方法来增量学习嵌入空间域的网络架构,使仔细选择的架构进行评估,在压缩架构搜索。给定一个教师网络,我们使用贝叶斯优化(BO)搜索一个压缩的网络架构,并在我们提出的嵌入空间上定义一个核函数来选择用于评估的架构。我们证明了我们的搜索算法可以显著优于各种基线方法,例如随机搜索和强化学习(Ashok et al.,2018年)。通过我们的方法发现的压缩架构也优于最先进的手动设计的紧凑架构ShuffleNet(Zhang等人,2018年)。我们还证明了学习的嵌入空间可以转移到新的架构搜索设置,例如更大的教师网络或不同架构家族中的教师网络,而无需任何训练。代码可在此处公开获取:https://github.com/Friedrich1006/ESNAC。
BTC-LLM: Efficient Sub-1-Bit LLM Quantization via Learnable Transformation and Binary Codebook
The Hong Kong University of Science and Technology(香港科技大学);City University of Hong Kong(香港城市大学);University of Electronic Science and Technology of China(电子科技大学)
Task-Oriented Feature Compression for Multimodal Understanding via Device-Edge Co-Inference
Institute of Artificial Intelligence (TeleAI) of China Telecom(中国电信人工智能研究所);School of Electronic and Information Engineering, Harbin Institute of Technology(哈尔滨工业大学电子与信息工程学院);Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology(香港科技大学电子与计算机工程系);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院)
CoVisco: Codec-Native Vision Encoder with Native Token Compression for Unified Image-Video Understanding
ERNIE Team, Baidu Inc.(百度公司ERNIE团队);The Hong Kong University of Science and Technology(香港科技大学);Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所)