Channel Exchanging Networks for Multimodal and Multitask Dense Image Prediction
Tsinghua University(清华大学);Renmin University of China(中国人民大学);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);JD.com Inc.(京东集团);JD Explore Academy(京东探索研究院)
Yikai Wang, Fuchun Sun, Wenbing Huang, Fengxiang He, Dacheng Tao
Multimodal fusion and multitask learning are two vital topics in machine learning. Despite the fruitful progress, existing methods for both problems are still brittle to the same challenge -- it remains dilemmatic to integrate the common information across modalities (resp. tasks) meanwhile preserving the specific patterns of each modality (resp. task). Besides, while they are actually closely related to each other, multimodal fusion and multitask learning are rarely explored within the same methodological framework before. In this paper, we propose Channel-Exchanging-Network (CEN) which is self-adaptive, parameter-free, and more importantly, applicable for multimodal and multitask dense image prediction. At its core, CEN adaptively exchanges channels between subnetworks of different modalities. Specifically, the channel exchanging process is self-guided by individual channel importance that is measured by the magnitude of Batch-Normalization (BN) scaling factor during training. For the application of dense image prediction, the validity of CEN is tested by four different scenarios: multimodal fusion, cycle multimodal fusion, multitask learning, and multimodal multitask learning. Extensive experiments on semantic segmentation via RGB-D data and image translation through multi-domain input verify the effectiveness of CEN compared to state-of-the-art methods. Detailed ablation studies have also been carried out, which demonstrate the advantage of each component we propose. Our code is available at https://github.com/yikaiw/CEN.
Beijing National Research Center for Information Science and Technology (BNRist)(北京信息科学与技术国家研究中心(BNRist));Tsinghua University(清华大学);Tencent AI Lab(腾讯AI实验室)
MMSF: Multitask and Multimodal Supervised Framework for WSI Classification and Survival Analysis
University of Chinese Academy of Sciences(中国科学院大学);Shanghai Advanced Research Institute, Chinese Academy of Sciences(中国科学院上海先进研究院);Department of Radiology, Changhai Hospital(昌海医院放射科)
Weakly Supervised Multimodal Temporal Forgery Localization via Multitask Learning
School of Computer Science and Engineering, MoE Key Laboratory of Information Technology, Guangdong Province Key Laboratory of Information Security Technology, Sun Yat-sen University(计算机科学与工程学院、信息科技教育部重点实验室、广东省信息安全技术重点实验室、中山大学);State Key Laboratory of Mathematical Engineering and Advanced Computing(数学工程与先进计算国家重点实验室)
Multimodal Fake News Detection: MFND Dataset and Shallow-Deep Multitask Learning
School of Artificial Intelligence, Hebei University of Technology(河北工业大学人工智能学院);School of Computing and Information Technology, Great Bay University(大湾大学计算与信息学院);Guangdong Provincial Key Laboratory of Intelligent Information Processing & Shenzhen Key Laboratory of Media Security, Shenzhen University(广东省智能信息处理重点实验室 & 深圳市媒体安全重点实验室);Dongguan Key Laboratory for Intelligence and Information Technology(东莞智能信息科技重点实验室)
MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);The University of Hong Kong(香港大学);The University of Adelaide(阿德莱德大学);Zhejiang University(浙江大学);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
M$^3$GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation
Peng Cheng Laboratory(鹏城实验室);University of Chinese Academy of Sciences(中国科学院大学);WeChat, Tencent Inc(腾讯公司微信事业部);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学深圳校区)
Switch-Reasoner: Learn When to Think in Multitask Mixtures via Reinforcement Learning
Wuhan University(武汉大学);Xiaomi Inc(小米公司);Wuhan University of Technology(武汉理工大学);Nanyang Technological University(南洋理工大学);The Hong Kong University of Science and Technology(香港科技大学)
TokenCom: Vision-Language Model for Multimodal and Multitask Token Communications
Hunan Provincial Key Laboratory of Intelligent Computing and Language Information Processing, Hunan Normal University(湖南省级智能计算与语言信息处理重点实验室,湖南师范大学);School of Information Science and Engineering, Hunan Normal University(信息科学与工程学院,湖南师范大学);Changsha Social Laboratory of Artificial Intelligence, Hunan University of Technology and Business(长沙人工智能社会实验室,湖南工业大学);School of Computer Science, Hunan University of Technology and Business(计算机科学学院,湖南工业大学);Department of Computer Science, Brunel University London(伦敦布鲁内尔大学计算机科学系);National Mobile Communications Research Laboratory, Southeast University(东南大学国家移动通信研究中心);Department of Electrical and Computer Engineering, University of Houston(电子与计算机工程系,休斯顿大学)
Specialty-Oriented Generalist Medical AI for Chest CT Screening
Rensselaer Polytechnic Institute(伦斯勒理工学院);Wake Forest University School of Medicine(维克森林大学医学院);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院)
School of Artificial Intelligence, UCAS(人工智能学院,UCAS);State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(多模态人工智能系统国家重点实验室,CASIA);Centre for Artificial Intelligence and Robotics, HKISI-CAS(人工智能与机器人中心,HKISI-CAS);Huawei Inc(华为公司)
Flood-DamageSense: Multimodal Mamba with Multitask Learning for Building Flood Damage Assessment using SAR Remote Sensing Imagery
Urban Resilience.AI Lab(城市韧性人工智能实验室);Zachry Department of Civil and Environmental Engineering(扎克里土木与环境工程系);Texas A&M University(德克萨斯A&M大学);College Station, TX(学院站, 德克萨斯)
Multimodal Prompt Perceiver: Empower Adaptiveness, Generalizability and Fidelity for All-in-One Image Restoration
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);University of Science and Technology of China(中国科学技术大学)
M3S-Net: Multimodal Feature Fusion Network Based on Multi-scale Data for Ultra-short-term PV Power Forecasting
School of Artificial Intelligence, Hebei University of Technology(人工智能学院,河北工业大学);University of Southern Queensland(南方昆士兰大学);School of Information Engineering, Tianjin University of Commerce(信息工程学院,天津商业大学);Hebei Province Key Laboratory of Big Data Calculation, Hebei University of Technology(大数据计算河北重点实验室,河北工业大学);General AI Lab, School of Engineering, Westlake University(通用人工智能实验室,西湖大学);School of Business and Law, Edith Cowan University(商学院和法学院,埃迪斯科文大学)