arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.10074cs.CVcs.SYeess.SY
收藏

VLF-MSC:基于视觉-语言特征的多模态语义通信系统

VLF-MSC: Vision-Language Feature-Based Multimodal Semantic Communication System

Gwangyeon Ahn, Jiwan Seo, Joonhyuk Kang

AI总结:

针对现有多模态语义通信分模态处理的不足,提出VLF-MSC系统,通过传输统一视觉-语言语义特征支持接收端图文双生成,在低信噪比下实现更高语义准确率与更低带宽占用。

摘要:

我们提出了基于视觉-语言特征的多模态语义通信系统VLF-MSC,这一统一系统可传输单一紧凑的视觉-语言表示,支持接收端同时完成图像和文本生成任务。与现有分模态处理的语义通信技术不同,VLF-MSC采用预训练视觉-语言模型(VLM)将源图像编码为视觉-语言语义特征(VLF),通过无线信道传输该特征。在接收端,基于解码器的语言模型和基于扩散模型的图像生成器均以VLF为条件,生成描述文本与语义对齐的图像。这种统一表示无需设置模态专属传输流或重传机制,提升了频谱效率与适配性。通过利用基础模型,该系统在保留语义保真度的同时,对信道噪声具备鲁棒性。实验表明,VLF-MSC性能优于纯文本和纯图像基线方法,在低信噪比(SNR)条件下两种模态的语义准确率更高,且带宽占用显著降低。

英文摘要:

We propose Vision-Language Feature-based Multimodal Semantic Communication (VLF-MSC), a unified system that transmits a single compact vision-language representation to support both image and text generation at the receiver. Unlike existing semantic communication techniques that process each modality separately, VLF-MSC employs a pre-trained vision-language model (VLM) to encode the source image into a vision-language semantic feature (VLF), which is transmitted over the wireless channel. At the receiver, a decoder-based language model and a diffusion-based image generator are both conditioned on the VLF to produce a descriptive text and a semantically aligned image. This unified representation eliminates the need for modality-specific streams or retransmissions, improving spectral efficiency and adaptability. By leveraging foundation models, the system achieves robustness to channel noise while preserving semantic fidelity. Experiments demonstrate that VLF-MSC outperforms text-only and image-only baselines, achieving higher semantic accuracy for both modalities under low SNR with significantly reduced bandwidth.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑