Controllable text-to-image (T2I) diffusion models generate images conditioned on both text prompts and semantic inputs of other modalities like edge maps. Nevertheless, current controllable T2I methods commonly face challenges related to efficiency and faithfulness, especially when conditioning on multiple inputs from either the same or diverse modalities. In this paper, we propose a novel Flexible and Efficient method, FlexEControl, for controllable T2I generation. At the core of FlexEControl is a unique weight decomposition strategy, which allows for streamlined integration of various input types. This approach not only enhances the faithfulness of the generated image to the control, but also significantly reduces the computational overhead typically associated with multimodal conditioning. Our approach achieves a reduction of 41% in trainable parameters and 30% in memory usage compared with Uni-ControlNet. Moreover, it doubles data efficiency and can flexibly generate images under the guidance of multiple input conditions of various modalities.
Multimodal Image Synthesis and Editing: The Generative AI Era
Max Planck Institute for Informatics(马克斯·普朗克信息研究所);S-Lab, Nanyang Technological University(南洋理工大学S实验室);Nanyang Technological University(南洋理工大学);Carnegie Mellon University(卡内基梅隆大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Great Bay University(大湾区大学);Nanyang Technological University(南洋理工大学);University of Oulu(奥卢大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Shenzhen University(深圳大学)
Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping
School of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学人工智能学院);Shanghai AI Laboratory(上海人工智能实验室);Tongji University(同济大学);The University of Hong Kong(香港大学)
OmniGen: Unified Multimodal Sensor Generation for Autonomous Driving
Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);Westlake University(西湖大学);Li Auto Inc.(Li汽车公司);The University of Toronto(多伦多大学);University of Macau(澳门大学)
Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performance
Shanghai AI Laboratory(上海人工智能实验室);Fudan University(复旦大学);Tsinghua University(清华大学);SenseTime Research(商汤科技研究院);Nanjing University(南京大学);The Chinese University of Hong Kong(香港中文大学);Shanghai Jiao Tong University(上海交通大学)
Multi-FRuGaL: Multimodal Flexible Redundancy-aware Decomposed Gated Learning for Cancer Diagnosis and Prognosis
Division of Computational Pathology, Department of Pathology and Laboratory Medicine, Indiana University School of Medicine(计算病理学部,病理学与实验室医学部,印第安纳大学医学院);IU Melvin and Bren Simon Comprehensive Cancer Center(印第安纳大学Melvin和Bren Simon综合癌症中心);Departments of Biostatistics and Health Data Science(生物统计学与健康数据科学部);Radiology and Imaging Sciences(放射学与影像科学部);Neurological Surgery(神经外科);Indiana University School of Medicine(印第安纳大学医学院);Department of Computer Science, Luddy School of Informatics, Computing, and Engineering(计算机科学部,Luddy信息、计算与工程学院)
Adaptive Look, Flexible Thought: Learning How and How Long to Think in Multimodal Reasoning
Sun Yat-sen University(中山大学);Shenzhen Loop Area Institute(深圳河套学院);Beijing Zhongguancun Academy(北京中关村研究院);Huawei Technologies Co., Ltd(华为技术有限公司);Harbin Institute of Technology(哈尔滨工业大学)
Flexible Multimodal Neuroimaging Fusion for Alzheimer's Disease Progression Prediction
Department of Computer Science and Engineering(计算机科学与工程系);Department of Biomedical Informatics(生物医学信息学系);Department of Neurology(神经病学系);Translational Data Analytics Institute(转化数据分析研究所)
ParaUni: Enhance Generation in Unified Multimodal Model with Reinforcement-driven Hierarchical Parallel Information Interaction
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China(中国科学技术大学,教育部类脑智能感知与认知重点实验室);Huawei Inc.(华为技术有限公司)
Universal Multimodal Representation for Language Understanding
Shanghai Jiao Tong University(上海交通大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));National Institute of Information and Communications Technology(日本国立信息通信研究机构)
Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referring
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所基础模型研究中心);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Peng Cheng Laboratory, Shenzhen, China(鹏城实验室);Wuhan AI Research, Wuhan, China(武汉人工智能研究所)
提出统一高分辨率通用模型 Griffon v2,通过轻量下采样投影器和即插即用视觉分词器实现视觉-语言共指代,在多项视觉感知任务上达到 SOTA 并超越专家模型。
Multimodal Personality Recognition using Cross-Attention Transformer and Behaviour Encoding
INRIA(法国国家信息与自动化研究所);Indian Institute of Information Technology, Allahabad(印度阿拉哈巴德信息技术学院);Birla Institute of Technology and Science, Pilani(印度皮拉尼贝拉技术科学学院)
Multimodal Task-Driven Dictionary Learning for Image Classification
Pennsylvania State University(宾夕法尼亚州立大学);Army Research Laboratory(美国陆军研究实验室);Bosch Research and Technology Center(博世研究与技术中心);West Virginia University(西弗吉尼亚大学)
Beyond Patches: Global-aware Autoregressive Model for Multimodal Few-Shot Font Generation
Wangxuan Institute of Computer Technology, Peking University(北京大学王学轩计算机技术研究所);School of Electronics Engineering and Computer Science, Peking University(北京大学电子工程与计算机科学学院)
Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)(中国科学院智能信息处理重点实验室);Key Laboratory of AI Safety, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学)