ParaUni: Enhance Generation in Unified Multimodal Model with Reinforcement-driven Hierarchical Parallel Information Interaction
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China(中国科学技术大学,教育部类脑智能感知与认知重点实验室);Huawei Inc.(华为技术有限公司)
Jiangtong Tan, Lin Liu, Jie Huanng, Xiaopeng Zhang, Qi Tian, Feng Zhao
Unified multimodal models significantly improve visual generation by combining vision-language models (VLMs) with diffusion models. However, existing methods struggle to fully balance sufficient interaction and flexible implementation due to vast representation difference. Considering abundant and hierarchical information in VLM's layers from low-level details to high-level semantics, we propose \textbf{ParaUni}. It extracts features from variants VLM's layers in a \textbf{Para}llel way for comprehensive information interaction and retains a flexible separation architecture to enhance generation in \textbf{Uni}fied multimodal model. Concretely, visual features from all VLM's layers are fed in parallel into a Layer Integration Module (LIM), which efficiently integrates fine-grained details and semantic abstractions and provides the fused representation as a condition to the diffusion model. To further enhance performance, we reveal that these hierarchical layers respond unequally to different rewards in Reinforcement Learning (RL). Crucially, we design a Layer-wise Dynamic Adjustment Mechanism (LDAM) to facilitate multiple reward improvements that aligns the hierarchical properties of these layers using RL. Extensive experiments show ParaUni leverages complementary multi-layer features to substantially improve generation quality and shows strong potential for multiple reward advances during RL stages. Code is available at https://github.com/JosephTiTan/ParaUni.
HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Grounding
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Pengcheng Laboratory(鹏城实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
ABMAMBA: Multimodal Large Language Model with Aligned Hierarchical Bidirectional Scan for Efficient Video Captioning
Keio University(庆应义塾大学);National Institute of Informatics(国立信息学研究所);National Institute of Informatics Research and Development Center for Large Language Models(国立信息学研究所大型语言模型研发中心)
Hierarchical Refinement of Universal Multimodal Attacks on Vision-Language Models
School of Electrical Engineering and Computer Science, the University of Queensland(电气工程与计算机科学学院,昆士兰大学);Department of Computer Science, City University of Hong Kong(计算机科学系,香港城市大学)
HMR3D: Hierarchical Multimodal Representation for 3D Scene Understanding with Large Vision-Language Model
Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore(高性能计算研究所,科技研究局,新加坡);Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore(前沿人工智能研究中心,科技研究局,新加坡);College of Computing and Data Science, Nanyang Technological University, Singapore(计算与数据科学学院,南洋理工大学,新加坡)
GeoBench: Rethinking Multimodal Geometric Problem-Solving via Hierarchical Evaluation
Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Fudan University(复旦大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding
Department of Computer Science, Khalifa University of Science and Technology(卡利法科技大学计算机科学系);Information Technology University(信息技术大学);KAU(卡乌大学);University of the Western Australia(西澳大学)
Test-time Adaptive Hierarchical Co-enhanced Denoising Network for Reliable Multimodal Classification
The Guangdong Provincial Key Laboratory of Computational Intelligence and Cyberspace Information, the School of Computer Science and Engineering, South China University of Technology(广东省计算智能与网络信息重点实验室、计算机科学与工程学院、华南理工大学);The Pazhou Laboratory(琶洲实验室)
AdaMHF: Adaptive Multimodal Hierarchical Fusion for Survival Prediction
School of Computing and Information Technology, Great Bay University(大湾区大学计算与信息技术学院);Harbin Institute of Technology(哈尔滨工业大学);Beihang University(北京航空航天大学);Macao Polytechnic University(澳门理工大学);Dongguan Key Laboratory for Intelligence and Information Technology(东莞市智能与信息技术重点实验室);Shenzhen Key Laboratory of Visual Object Detection and Recognition, Harbin Institute of Technology Shenzhen(哈尔滨工业大学深圳校区深圳市视觉目标检测与识别重点实验室)
Hierarchical Prototype-based Domain Priors for Multiple Instance Learning in Multimodal Histopathology Analysis
College of Computer and Information Science, Fujian Agriculture and Forestry University(福建农林大学计算机与信息科学学院);College of Future Technology, Fujian Agriculture and Forestry University(福建农林大学未来技术学院);Digital Fujian Institute of Agricultural Big Data, Fujian Agriculture and Forestry University(福建农业大数据数字福建研究院);Department of Pathology, Clinical Oncology School of Fujian Medical University, and Fujian Cancer Hospital(福建医科大学临床肿瘤学院病理科及福建癌症医院)
Decoupled Hierarchical Distillation for Multimodal Emotion Recognition
School of Computer Science and Engineering, and the Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University(计算机科学与工程学院,新一代人工智能技术及跨学科应用重点实验室,东南大学);School of Computer Science and Engineering, Nanjing University of Science and Technology(计算机科学与工程学院,南京理工大学);Institute of Intelligent Information Processing, Taizhou University(智能信息处理研究院,台州大学);School of Engineering Science, University of Chinese Academy of Sciences(工程科学学院,中国科学院大学);Peng Cheng Laboratory(鹏城实验室);School of Computer Science and Engineering, Nanyang Technological University(计算机科学与工程学院,南洋理工大学)
Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning
Academy for Engineering and Technology, Fudan University(复旦大学工程与技术学院);Cognition and Intelligent Technology Laboratory(认知与智能技术实验室);Institute of Metaverse & Intelligent Medicine, Fudan University(复旦大学元宇宙与智能医学研究院);Jilin Provincial Key Laboratory of Intelligence Science and Engineering(吉林省智能科学与工程重点实验室);Engineering Research Center of AI and Robotics, Ministry of Education(教育部人工智能与机器人工程研究中心)
DashFusion: Dual-stream Alignment with Hierarchical Bottleneck Fusion for Multimodal Sentiment Analysis
School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院);Beijing National Research Center for Information Science and Technology, Tsinghua University(清华大学信息科学与技术国家研究中心);Department of Automation, Tsinghua University(清华大学自动化系)
Multimodal Cancer Survival Analysis via Hypergraph Learning with Cross-Modality Rebalance
Faculty of Computing, Harbin Institute of Technology(哈尔滨工业大学计算机学院);School of Software, Tsinghua University(清华大学软件学院);School of Computer Science and Engineering, UNSW Sydney(新南威尔士大学计算机科学与工程学院)