arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.22112cs.MM
收藏

面向生成式音频驱动视频会议的多模态语义通信

Multimodal Semantic Communication for Generative Audio-Driven Video Conferencing

Haonan Tong, Haopeng Li, Hongyang Du, Zhaohui Yang, Changchuan Yin, Dusit Niyato

AI总结:

提出 Wav2Vid 多模态语义通信框架,利用全时长音频与短视频语义及 GAN 生成说话人头部视频,在保持感知质量的同时将传输数据量最高减少 83%。

摘要:

本文研究了一种用于视频会议的高效多模态数据通信方案。在我们所考虑的系统中,演讲者向观众发表讲话,同时传输说话人头部视频和音频。由于演讲者不会频繁改变姿势,并且音频(语音和音乐)需要高保真传输,因此视频中存在冗余的视觉数据,可以通过根据音频生成视频来去除。为此,我们提出了 wave-to-video(Wav2Vid)系统,这是一种高效的视频传输框架,通过根据音频生成说话人头部视频来减少传输数据量。具体而言,全时长音频和短时长视频数据通过无线信道同步传输,并由神经网络(NNs)提取和编码音频与视频语义。随后,接收端结合解码后的音频和视频数据,并使用基于生成对抗网络(GAN)的模型生成演讲者的唇部运动视频。仿真结果表明,所提出的 Wav2Vid 系统可将传输数据量减少最高达 83%,同时保持所生成会议视频的感知质量。

英文摘要:

This paper studies an efficient multimodal data communication scheme for video conferencing. In our considered system, a speaker gives a talk to the audiences, with talking head video and audio being transmitted. Since the speaker does not frequently change posture and high-fidelity transmission of audio (speech and music) is required, redundant visual video data exists and can be removed by generating the video from the audio. To this end, we propose a wave-to-video (Wav2Vid) system, an efficient video transmission framework that reduces transmitted data by generating talking head video from audio. In particular, full-duration audio and short-duration video data are synchronously transmitted through a wireless channel, with neural networks (NNs) extracting and encoding audio and video semantics. The receiver then combines the decoded audio and video data, as well as uses a generative adversarial network (GAN) based model to generate the lip movement videos of the speaker. Simulation results show that the proposed Wav2Vid system can reduce the amount of transmitted data by up to 83% while maintaining the perceptual quality of the generated conferencing video.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑