使用关键点提取和数字木偶减少视频流的延迟和带宽
Reducing latency and bandwidth for video streaming using keypoint extraction and digital puppetry
摘要:
COVID-19使视像通信成为最重要的信息交流方式之一。虽然已经对视频流管道的优化进行了广泛的研究,特别是开发新的视频编解码器,但需要进一步改善视频质量和延迟,特别是在较差的网络条件下。本文提出了一种替代传统的编解码器,通过实现一个关键点为中心的编码器依赖于关键点信息的传输从视频源。解码器使用流式传输的关键点来生成保留输入馈送中的语义特征的重构。针对视频通话应用,我们检测并通过网络传输人体姿态和人脸网格信息,这些信息以动画木偶的形式显示在接收端。使用高效的姿势和人脸网格检测结合基于cannon的动画,我们展示了一个原型,需要低于35 kbps的带宽,一个数量级的减少,在典型的视频通话系统。在标准笔记本电脑上,由于网格提取和动画而增加的计算延迟低于120 ms,展示了该框架在实时应用中的潜力。这个工作的代码可以在https://github.com/shubhamchandak94/digital-puppetry/上找到。
英文摘要:
COVID-19 has made video communication one of the most important modes of information exchange. While extensive research has been conducted on the optimization of the video streaming pipeline, in particular the development of novel video codecs, further improvement in the video quality and latency is required, especially under poor network conditions. This paper proposes an alternative to the conventional codec through the implementation of a keypoint-centric encoder relying on the transmission of keypoint information from within a video feed. The decoder uses the streamed keypoints to generate a reconstruction preserving the semantic features in the input feed. Focusing on video calling applications, we detect and transmit the body pose and face mesh information through the network, which are displayed at the receiver in the form of animated puppets. Using efficient pose and face mesh detection in conjunction with skeleton-based animation, we demonstrate a prototype requiring lower than 35 kbps bandwidth, an order of magnitude reduction over typical video calling systems. The added computational latency due to the mesh extraction and animation is below 120ms on a standard laptop, showcasing the potential of this framework for real-time applications. The code for this work is available at https://github.com/shubhamchandak94/digital-puppetry/.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。