Recently, generative adversarial networks (GANs) have shown promising performance in generating realistic images. However, they often struggle in learning complex underlying modalities in a given dataset, resulting in poor-quality generated images. To mitigate this problem, we present a novel approach called mixture of experts GAN (MEGAN), an ensemble approach of multiple generator networks. Each generator network in MEGAN specializes in generating images with a particular subset of modalities, e.g., an image class. Instead of incorporating a separate step of handcrafted clustering of multiple modalities, our proposed model is trained through an end-to-end learning of multiple generators via gating networks, which is responsible for choosing the appropriate generator network for a given condition. We adopt the categorical reparameterization trick for a categorical decision to be made in selecting a generator while maintaining the flow of the gradients. We demonstrate that individual generators learn different and salient subparts of the data and achieve a multiscale structural similarity (MS-SSIM) score of 0.2470 for CelebA and a competitive unsupervised inception score of 8.33 in CIFAR-10.
MMoT: Mixture-of-Modality-Tokens Transformer for Composed Multimodal Conditional Image Synthesis
South China University of Technology（华南理工大学）；JD Explore Academy（京东探索研究院）；Nanyang Technological University（南洋理工大学）；Shanghai Jiao Tong University（上海交通大学）
Future prediction is a fundamental principle of intelligence that helps plan actions and avoid possible dangers. As the future is uncertain to a large extent, modeling the uncertainty and multimodality of the future states is of great relevance. Existing approaches are rather limited in this regard and mostly yield a single hypothesis of the future or, at the best, strongly constrained mixture components that suffer from instabilities in training and mode collapse. In this work, we present an approach that involves the prediction of several samples of the future with a winner-takes-all loss an…
Large sparsely-activated models have obtained excellent performance in multiple domains. However, such models are typically trained on a single modality at a time. We present the Language-Image MoE, LIMoE, a sparse mixture of experts model capable of multimodal learning. LIMoE accepts both images and text simultaneously, while being trained using a contrastive loss. MoEs are a natural fit for a multimodal backbone, since expert layers can learn an appropriate partitioning of modalities. However, new challenges arise; in particular, training stability and balanced expert utilization, for which …
Equipping predicted segmentation with calibrated uncertainty is essential for safety-critical applications. In this work, we focus on capturing the data-inherent uncertainty (aka aleatoric uncertainty) in segmentation, typically when ambiguities exist in input images. Due to the high-dimensional output space and potential multiple modes in segmenting ambiguous images, it remains challenging to predict well-calibrated uncertainty for segmentation. To tackle this problem, we propose a novel mixture of stochastic experts (MoSE) model, where each expert network estimates a distinct mode of the ale…
Mixture-of-Thought-Tokens: Unifying Perception and Reasoning for Free-form Multimodal Grounding
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所）；Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd（星辰通用人工智能实验室，中国电信人工智能技术（北京）有限公司）；University of Science and Technology Beijing（北京科技大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Shanghai Jiao Tong University（上海交通大学）
Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
Department of Computer Science and Technology, BNRist, Tsinghua University, Beijing, China（计算机科学与技术系，BNRist，清华大学，北京，中国）；Alibaba Group, Hangzhou, China（阿里巴巴集团，杭州，中国）
Zhejiang University-University of Illinois at Urbana-Champaign Institute（浙江大学-伊利诺伊大学厄巴纳香槟分校联合学院）；Zhejiang University（浙江大学）
Multimodal sentiment analysis (MSA) is an important way of observing mental activities with the help of data captured from multiple modalities. However, due to the recording or transmission error, some modalities may include incomplete data. Most existing works that address missing modalities usually assume a particular modality is completely missing and seldom consider a mixture of missing across multiple modalities. In this paper, we propose a simple yet effective meta-sampling approach for multimodal sentiment analysis with missing modalities, namely Missing Modality-based Meta Sampling (M3…
ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
NepAl Applied Mathematics and Informatics Institute（尼泊尔应用数学与信息学研究所）；Fogsphere (Redev.AI Ltd)（福格球（Redev.AI有限公司））；University of Lausanne（洛桑大学）；West Virginia University（西弗吉尼亚大学）；University of Verona（维罗纳大学）；University College London（伦敦大学学院）；University of Aberdeen（阿伯丁大学）
Predicting Immune Biomarkers with MultiModal Mixture-of-Expert Pathology Foundation Models Empowers Precision Oncology
Program of Computational Biology and Bioinforamtics, Yale University（耶鲁大学计算生物学与生物信息学项目）；Broad Institute of MIT and Harvard（麻省理工学院与哈佛大学博德研究所）；Department of Statistics and Data Science, Northwestern University（西北大学统计与数据科学系）；Department of Computer Science, Northeastern University（东北大学计算机科学系）；Department of Computer Science, Harvard University（哈佛大学计算机科学系）；Department of Pathology, Yale University（耶鲁大学病理学系）；Department of Anatomic Pathology and Laboratory Medicine, Hospital of the University of Pennsylvania（宾夕法尼亚大学医院解剖病理学与检验医学系）；Department of Pathology and Laboratory Medicine, University of California, San Francisco（加州大学旧金山分校病理学与检验医学系）；Department of Pathology and Laboratory Medicine, KK Women’s and Children’s Hospital（竹脚妇幼医院病理学与检验医学系）；Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania（宾夕法尼亚大学佩雷尔曼医学院生物统计学、流行病学与信息学系）
MOVE: A Mixture-of-Vision-Encoders Approach for Domain-Focused Vision-Language Processing
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）；Artificial Intelligence Research Institute (AIRI)（人工智能研究院）；Higher School of Economics（高等经济学院）
FlexiReID: Adaptive Mixture of Expert for Multi-Modal Person Re-Identification
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（多媒体可信感知与高效计算重点实验室，中国教育部，厦门大学）；National University of Singapore（新加坡国立大学）；Tencent YouTu Lab（腾讯优图实验室）；Institute of Artificial Intelligence,Xiamen University（人工智能研究院，厦门大学）
DynFS-MoE: Dynamic Functional-Structural Mixture-of-Experts for Post-Traumatic Epilepsy Diagnosis
Department of Systems Engineering, Stevens Institute of Technology（史蒂文斯理工学院系统工程系）；Department of Neurosurgery, Robert Wood Johnson Medical School, Rutgers University（罗格斯大学罗伯特·伍德·约翰逊医学院神经外科）
$γ-$MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large Language Models
Technical University Of Denmark（丹麦技术大学）；OpenGVLab（开放视觉实验室）；Shanghai AI Laboratory（上海人工智能实验室）；Xiamen University（厦门大学）；National University of Singapore（新加坡国立大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）
MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learning
Brown University（布朗大学）；Salesforce AI Research（Salesforce AI研究院）；NVIDIA Research（英伟达研究院）；Carnegie Mellon University（卡内基梅隆大学）；Princeton University（普林斯顿大学）；Texas A&M University（德克萨斯A&M大学）；California Institute of Technology（加州理工学院）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）
Generalist-Specialist Mixture-of-Experts for Rare Pathology Detection in Multimodal Imaging
Technical University of Munich (TUM)（慕尼黑工业大学）；TUM University Hospital（慕尼黑工业大学附属医院）；Hasso Plattner Institute for Digital Engineering（哈索·普拉特纳数字工程研究所）；University of Potsdam（波茨坦大学）；Imperial College London（伦敦帝国理工学院）
Multi-level Mixture of Experts for Multimodal Entity Linking
School of Computer；Information Technology Shanxi University Taiyuan China；School of Computer Science；Informatics Cardiff University Cardiff UK；ILCC, School of Informatics University of Edinburgh Edinburgh UK；Information Technology Shanxi University；Informatics Cardiff University；ILCC, School of Informatics University of Edinburgh
MoTE: Mixture of Ternary Experts for Memory-efficient Large Multimodal Models
Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院人工智能安全重点实验室，计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering
School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；Shanghai Open University（上海开放大学）；ZhuQingTing Data Technology (Zhejiang) Co., Ltd（朱清庭数据科技（浙江）有限公司）
MoPE: Mixture of Prompt Experts for Parameter-Efficient and Scalable Multimodal Fusion
Department of Computing, The Hong Kong Polytechnic University（计算系，香港理工大学）；Research Institute of Multiple Agents and Embodied Intelligence, Pengcheng Laboratory（多智能体与具身智能研究院，鹏城实验室）
The problem of multimodal clustering arises whenever the data are gathered with several physically different sensors. Observations from different modalities are not necessarily aligned in the sense there there is no obvious way to associate or to compare them in some common space. A solution may consist in considering multiple clustering tasks independently for each modality. The main difficulty with such an approach is to guarantee that the unimodal clusterings are mutually consistent. In this paper we show that multimodal clustering can be addressed within a novel framework, namely conjugate…
