大型AI模型赋能的多模态语义通信
Large AI Model Empowered Multimodal Semantic Communications
AI总结：
针对多模态语义通信中的异构、歧义和信道失真问题，提出LAM-MSC框架，融合MLM对齐、LLM个性化知识库与CGE信道估计，仿真验证其性能优越。
摘要：
包括文本、音频、图像和视频在内的多模态信号可以被整合到语义通信（Semantic Communication，SC）系统中，以在语义层面提供低时延、高质量的沉浸式体验。然而，多模态SC面临若干挑战，包括数据异构性、语义歧义以及传输过程中的信号失真。大型AI模型的最新进展，尤其是在多模态语言模型（Multimodal Language Model，MLM）和大型语言模型（Large Language Model，LLM）方面的进展，为解决这些问题提供了潜在方案。为此，我们提出了一种基于大型AI模型的多模态SC（Large AI Model-based Multimodal SC，LAM-MSC）框架。在该框架中，我们首先提出基于MLM的多模态对齐（MLM-based Multimodal Alignment，MMA），利用MLM实现多模态数据与单模态数据之间的转换，同时保持语义一致性。随后，提出个性化的基于LLM的知识库（LLM-based Knowledge Base，LKB），使用户能够通过LLM执行个性化语义提取或恢复，从而有效解决语义歧义。最后，我们采用基于条件生成对抗网络的信道估计（Conditional Generative adversarial network-based channel Estimation，CGE）来估计无线信道状态信息。该方法有效缓解了SC中衰落信道的影响。最后，我们进行了仿真，结果证明了LAM-MSC框架的优越性能。
英文摘要：
Multimodal signals, including text, audio, image, and video, can be integrated into Semantic Communication (SC) systems to provide an immersive experience with low latency and high quality at the semantic level. However, the multimodal SC has several challenges, including data heterogeneity, semantic ambiguity, and signal distortion during transmission. Recent advancements in large AI models, particularly in the Multimodal Language Model (MLM) and Large Language Model (LLM), offer potential solutions for addressing these issues. To this end, we propose a Large AI Model-based Multimodal SC (LAM-MSC) framework, where we first present the MLM-based Multimodal Alignment (MMA) that utilizes the MLM to enable the transformation between multimodal and unimodal data while preserving semantic consistency. Then, a personalized LLM-based Knowledge Base (LKB) is proposed, which allows users to perform personalized semantic extraction or recovery through the LLM. This effectively addresses the semantic ambiguity. Finally, we apply the Conditional Generative adversarial network-based channel Estimation (CGE) for estimating the wireless channel state information. This approach effectively mitigates the impact of fading channels in SC. Finally, we conduct simulations that demonstrate the superior performance of the LAM-MSC framework.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
