TokenCom: Vision-Language Model for Multimodal and Multitask Token Communications
Hunan Provincial Key Laboratory of Intelligent Computing and Language Information Processing, Hunan Normal University(湖南省级智能计算与语言信息处理重点实验室,湖南师范大学);School of Information Science and Engineering, Hunan Normal University(信息科学与工程学院,湖南师范大学);Changsha Social Laboratory of Artificial Intelligence, Hunan University of Technology and Business(长沙人工智能社会实验室,湖南工业大学);School of Computer Science, Hunan University of Technology and Business(计算机科学学院,湖南工业大学);Department of Computer Science, Brunel University London(伦敦布鲁内尔大学计算机科学系);National Mobile Communications Research Laboratory, Southeast University(东南大学国家移动通信研究中心);Department of Electrical and Computer Engineering, University of Houston(电子与计算机工程系,休斯顿大学)
Feibo Jiang, Siwei Tu, Li Dong, Xiaolong Li, Kezhi Wang, Cunhua Pan, Zhu Han, Jiangzhou Wang
Visual-Language Models (VLMs), with their strong capabilities in image and text understanding, offer a solid foundation for intelligent communications. However, their effectiveness is constrained by limited token granularity, overlong visual token sequences, and inadequate cross-modal alignment. To overcome these challenges, we propose TaiChi, a novel VLM framework designed for token communications. TaiChi adopts a dual-visual tokenizer architecture that processes both high- and low-resolution images to collaboratively capture pixel-level details and global conceptual features. A Bilateral Attention Network (BAN) is introduced to intelligently fuse multi-scale visual tokens, thereby enhancing visual understanding and producing compact visual tokens. In addition, a Kolmogorov Arnold Network (KAN)-based modality projector with learnable activation functions is employed to achieve precise nonlinear alignment from visual features to the text semantic space, thus minimizing information loss. Finally, TaiChi is integrated into a multimodal and multitask token communication system equipped with a joint VLM-channel coding scheme. Experimental results validate the superior performance of TaiChi, as well as the feasibility and effectiveness of the TaiChi-driven token communication system.
MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);The University of Hong Kong(香港大学);The University of Adelaide(阿德莱德大学);Zhejiang University(浙江大学);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
Switch-Reasoner: Learn When to Think in Multitask Mixtures via Reinforcement Learning
Wuhan University(武汉大学);Xiaomi Inc(小米公司);Wuhan University of Technology(武汉理工大学);Nanyang Technological University(南洋理工大学);The Hong Kong University of Science and Technology(香港科技大学)
MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation
Centre for Perceptual and Interactive Intelligence, the Chinese University of Hong Kong(香港中文大学感知与交互智能中心);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);SenseTime Research(商汤科技研究院);Department of Electronic Engineering, the Chinese University of Hong Kong(香港中文大学电子工程系)
Channel Exchanging Networks for Multimodal and Multitask Dense Image Prediction
Tsinghua University(清华大学);Renmin University of China(中国人民大学);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);JD.com Inc.(京东集团);JD Explore Academy(京东探索研究院)
CoLA: Cross-Modal Low-rank Adaptation for Multimodal Downstream Tasks
Centre for Vision, Speech and Signal Processing (CVSSP)(视觉、语音和信号处理中心);University of Surrey(塞维利亚大学);Surrey Institute for People-Centred AI(以人为本的人工智能研究所)
From Semantic to Token Communication: The Next Paradigm for Large-Model-Driven 6G Intelligent Connectivity
Beijing Institute of Technology(北京理工大学);Zhongguancun Academy(中关村学院);The University of Hong Kong(香港大学);University of Surrey(萨里大学);Shanghai Jiao Tong University(上海交通大学);Beijing University of Posts and Telecommunications(北京邮电大学);Southeast University(东南大学);University of Southampton(南安普顿大学);Ocean University of China(中国海洋大学);Singapore University of Technology and Design(新加坡科技设计大学)
MMSF: Multitask and Multimodal Supervised Framework for WSI Classification and Survival Analysis
University of Chinese Academy of Sciences(中国科学院大学);Shanghai Advanced Research Institute, Chinese Academy of Sciences(中国科学院上海先进研究院);Department of Radiology, Changhai Hospital(昌海医院放射科)
Weakly Supervised Multimodal Temporal Forgery Localization via Multitask Learning
School of Computer Science and Engineering, MoE Key Laboratory of Information Technology, Guangdong Province Key Laboratory of Information Security Technology, Sun Yat-sen University(计算机科学与工程学院、信息科技教育部重点实验室、广东省信息安全技术重点实验室、中山大学);State Key Laboratory of Mathematical Engineering and Advanced Computing(数学工程与先进计算国家重点实验室)
GeoChat: Grounded Large Vision-Language Model for Remote Sensing
Mohamed bin Zayed University of AI(穆罕默德·本·扎耶德人工智能大学);Birla Institute of Technology & Science, Hyderabad(海得拉巴贝拉理工学院);Australian National University(澳大利亚国立大学);Linköping University(林雪平大学)
A Versatile Pathology Co-pilot via Reasoning Enhanced Multimodal Large Language Model
Hong Kong University of Science and Technology(香港科技大学);Southern Medical University(南方医科大学);Nanfang Hospital and School of Basic Medical Sciences(南方医院和基础医学科学学院);Chinese University of Hong Kong(香港中文大学)
Multimodal Fake News Detection: MFND Dataset and Shallow-Deep Multitask Learning
School of Artificial Intelligence, Hebei University of Technology(河北工业大学人工智能学院);School of Computing and Information Technology, Great Bay University(大湾大学计算与信息学院);Guangdong Provincial Key Laboratory of Intelligent Information Processing & Shenzhen Key Laboratory of Media Security, Shenzhen University(广东省智能信息处理重点实验室 & 深圳市媒体安全重点实验室);Dongguan Key Laboratory for Intelligence and Information Technology(东莞智能信息科技重点实验室)
M$^3$GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation
Peng Cheng Laboratory(鹏城实验室);University of Chinese Academy of Sciences(中国科学院大学);WeChat, Tencent Inc(腾讯公司微信事业部);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学深圳校区)
Specialty-Oriented Generalist Medical AI for Chest CT Screening
Rensselaer Polytechnic Institute(伦斯勒理工学院);Wake Forest University School of Medicine(维克森林大学医学院);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院)
Flood-DamageSense: Multimodal Mamba with Multitask Learning for Building Flood Damage Assessment using SAR Remote Sensing Imagery
Urban Resilience.AI Lab(城市韧性人工智能实验室);Zachry Department of Civil and Environmental Engineering(扎克里土木与环境工程系);Texas A&M University(德克萨斯A&M大学);College Station, TX(学院站, 德克萨斯)
School of Artificial Intelligence, UCAS(人工智能学院,UCAS);State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(多模态人工智能系统国家重点实验室,CASIA);Centre for Artificial Intelligence and Robotics, HKISI-CAS(人工智能与机器人中心,HKISI-CAS);Huawei Inc(华为公司)