arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.00482cs.CVcs.ITmath.IT
收藏

TokenCom: 一种用于多模态和多任务令牌通信的视觉-语言模型

TokenCom: Vision-Language Model for Multimodal and Multitask Token Communications

Hunan Provincial Key Laboratory of Intelligent Computing and Language Information Processing, Hunan Normal University(湖南省级智能计算与语言信息处理重点实验室,湖南师范大学);School of Information Science and Engineering, Hunan Normal University(信息科学与工程学院,湖南师范大学);Changsha Social Laboratory of Artificial Intelligence, Hunan University of Technology and Business(长沙人工智能社会实验室,湖南工业大学);School of Computer Science, Hunan University of Technology and Business(计算机科学学院,湖南工业大学);Department of Computer Science, Brunel University London(伦敦布鲁内尔大学计算机科学系);National Mobile Communications Research Laboratory, Southeast University(东南大学国家移动通信研究中心);Department of Electrical and Computer Engineering, University of Houston(电子与计算机工程系,休斯顿大学)

Feibo Jiang, Siwei Tu, Li Dong, Xiaolong Li, Kezhi Wang, Cunhua Pan, Zhu Han, Jiangzhou Wang

AI总结:

TokenCom提出了一种新的视觉-语言模型框架TaiChi,通过双视觉分词器和双向注意力网络提升多模态和多任务令牌通信的性能。

摘要:

视觉-语言模型(VLMs)凭借其在图像和文本理解方面的强大能力,为智能通信提供了坚实的基础。然而,其有效性受到令牌粒度有限、视觉令牌序列过长以及跨模态对齐不足的限制。为克服这些挑战,我们提出TaiChi,一种新的VLM框架,专门用于令牌通信。TaiChi采用双视觉分词器架构,处理高分辨率和低分辨率图像,以协同捕捉像素级细节和全局概念特征。引入双向注意力网络(BAN)以智能融合多尺度视觉令牌,从而增强视觉理解并生成紧凑的视觉令牌。此外,采用基于Kolmogorov Arnold网络(KAN)的模态投影器,具有可学习的激活函数,以实现从视觉特征到文本语义空间的精确非线性对齐,从而最小化信息损失。最后,TaiChi被整合到一个多模态和多任务令牌通信系统中,配备了联合VLM-信道编码方案。实验结果验证了TaiChi的优越性能,以及由TaiChi驱动的令牌通信系统的可行性和有效性。

英文摘要:

Visual-Language Models (VLMs), with their strong capabilities in image and text understanding, offer a solid foundation for intelligent communications. However, their effectiveness is constrained by limited token granularity, overlong visual token sequences, and inadequate cross-modal alignment. To overcome these challenges, we propose TaiChi, a novel VLM framework designed for token communications. TaiChi adopts a dual-visual tokenizer architecture that processes both high- and low-resolution images to collaboratively capture pixel-level details and global conceptual features. A Bilateral Attention Network (BAN) is introduced to intelligently fuse multi-scale visual tokens, thereby enhancing visual understanding and producing compact visual tokens. In addition, a Kolmogorov Arnold Network (KAN)-based modality projector with learnable activation functions is employed to achieve precise nonlinear alignment from visual features to the text semantic space, thus minimizing information loss. Finally, TaiChi is integrated into a multimodal and multitask token communication system equipped with a joint VLM-channel coding scheme. Experimental results validate the superior performance of TaiChi, as well as the feasibility and effectiveness of the TaiChi-driven token communication system.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑